Das deutsche Python-Forum

Code: Alles auswählen

import requests
from bs4 import BeautifulSoup
r = requests.get("http://www.sciencedirect.com/science/journal/0304405X")
soup = BeautifulSoup(r.content, 'lxml')
for oltag in soup.find_all('ol', {'class': 'articleList results'}):
    for ultag in soup.find_all('ul', {'class': 'article'}):
        for litag in ultag.find_all('li', {'class': 'authors '}):

Code: Alles auswählen

In [10]: import requests

In [11]: from bs4 import BeautifulSoup

In [12]: r = requests.get("http://www.sciencedirect.com/science/journal/0304405X
    ...: ")

In [13]: soup = BeautifulSoup(r.content, 'lxml')

In [14]: soup.find_all('ol', {'class': 'articleList results'})
Out[14]: []

Code: Alles auswählen

#!/usr/bin/env python3
# -*- coding: utf-8 -*-

import requests
from bs4 import BeautifulSoup

USERAGENT = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.133 Safari/537.36'

# Query
QUERY_HEADERS = {
    'User-Agent': USERAGENT
}

QUERY_URL = 'http://www.sciencedirect.com/science/journal/0304405X'

QUERY_PAYLOAD = {
    'pageMetaData': {
        'cid': '',
        'issn': '0304405X',
        'allowVIA': ''
    }
}

def main():
    r_session = requests.session()
    r = r_session.get(QUERY_URL, headers=QUERY_HEADERS, data=QUERY_PAYLOAD)
    soup = BeautifulSoup(r.text, 'html.parser')
    listQuery = soup.find_all('li', {'class': 'authors'})
    for intIndex, actual in enumerate(listQuery):
        print('Zeile: ' + str(intIndex) + ' Autor: ' + actual.text)

if __name__ == "__main__":
    main()

Code: Alles auswählen

Zeile: 0 Autor: 
Zeile: 1 Autor: Viral Acharya, Zhaoxia Xu
Zeile: 2 Autor: Sabrina Buti, Barbara Rindi, Ingrid M. Werner
Zeile: 3 Autor: Marco Di Maggio, Amir Kermani, Zhaogang Song
Zeile: 4 Autor: Cornelius Schmidt, Rüdiger Fahlenbrach
Zeile: 5 Autor: Benjamin Bennett, J. Carr Bettis, Radhakrishnan Gopalan, Todd Milbourn
Zeile: 6 Autor: Charles M.C. Lee, Eric C. So
Zeile: 7 Autor: Camelia M. Kuhnen, Andrei C. Miu
Zeile: 8 Autor: Byoung-Hyoun Hwang, Hugh Hoikwang Kim
Zeile: 9 Autor: Marti G. Subrahmanyam, Dragon Yongjun Tang, Sarah Qian Wang
Zeile: 10 Autor: Juan M. Londono, Hao Zhou