Das deutsche Python-Forum

Code: Alles auswählen

rom time import sleep
from time import time
from random import randint
from bs4 import BeautifulSoup
from requests import get
import pandas

#loop through individual webpages
pages = [str(i) for i in range(1,3)]


name = []
date = []
location = []

start_time = time()
requests = 0

for page in pages:

    url = 'https://www.eventbrite.com/d/malaysia--kuala-lumpur--85675181/all-events/?page=' + str(page)
    response = get(url)

    sleep(randint(1,3))

    requests += 1
    elapsed_time = time() - start_time
    print('Request: {}; Frequency: {} requests/s'.format(requests, requests/elapsed_time))

    if response.status_code != 200:
        print('Request: {}; Status Code: {}'.format(requests, response.status_code))

    html_soup = BeautifulSoup(response.text, 'html.parser')

    #main div
    event_containers = html_soup.find_all('div', class_ = 'eds-media-card-content__content__principal')

    for container in event_containers:

        #get event name
        event_name = container.h3.div.div.text
        name.append(event_name)

        #get event day & date
        event_date = container.div.div.text
        date.append(event_date)

        #get event location
        event_location = container.find('div', class_ = 'card-text--truncated__one')
        location.append(event_location)

event_list = pandas.DataFrame({
    'event': name,
    'date': date,
    'location': location})

event_list = event_list.drop_duplicates()    
print(event_list)

Code: Alles auswählen

Traceback (most recent call last):
  File "C:/Users/foo_the_user/.PyCharmEdu2019.2/config/scratches/scratch.py", line 4, in <module>
    from bs4 import BeautifulSoup
ModuleNotFoundError: No module named 'bs4'
Process finished with exit code 1

Code: Alles auswählen

Windows PowerShell
Copyright (C) Microsoft Corporation. Alle Rechte vorbehalten.

Lernen Sie das neue plattformübergreifende PowerShell kennen – https://aka.ms/pscore6

PS C:\WINDOWS\system32> pip intall bs4
ERROR: unknown command "intall" - maybe you meant "install"
PS C:\WINDOWS\system32> pip install bs4
Collecting bs4
  Downloading bs4-0.0.1.tar.gz (1.1 kB)
Requirement already satisfied: beautifulsoup4 in c:\program files\python37\lib\site-packages (from bs4) (4.8.2)
Requirement already satisfied: soupsieve>=1.2 in c:\program files\python37\lib\site-packages (from beautifulsoup4->bs4) (2.0)
Installing collected packages: bs4
    Running setup.py install for bs4 ... done
Successfully installed bs4-0.0.1
PS C:\WINDOWS\system32> pip install pandas
Collecting pandas
  Downloading pandas-1.0.3-cp37-cp37m-win_amd64.whl (8.7 MB)
     |████████████████████████████████| 8.7 MB 6.4 MB/s
Collecting pytz>=2017.2
  Downloading pytz-2019.3-py2.py3-none-any.whl (509 kB)
     |████████████████████████████████| 509 kB 6.4 MB/s
Collecting numpy>=1.13.3
  Downloading numpy-1.18.2-cp37-cp37m-win_amd64.whl (12.8 MB)
     |████████████████████████████████| 12.8 MB 6.8 MB/s
Collecting python-dateutil>=2.6.1
  Downloading python_dateutil-2.8.1-py2.py3-none-any.whl (227 kB)
     |████████████████████████████████| 227 kB 3.3 MB/s
Requirement already satisfied: six>=1.5 in c:\users\foo_the_user\appdata\roaming\python\python37\site-packages (from python-dateutil>=2.6.1->pandas) (1.12.0)
Installing collected packages: pytz, numpy, python-dateutil, pandas
Successfully installed numpy-1.18.2 pandas-1.0.3 python-dateutil-2.8.1 pytz-2019.3
PS C:\WINDOWS\system32> pip install time
ERROR: Could not find a version that satisfies the requirement time (from versions: none)
ERROR: No matching distribution found for time
PS C:\WINDOWS\system32> pip install request
Collecting request
  Downloading request-2019.4.13.tar.gz (1.3 kB)
Collecting get
  Downloading get-2019.4.13.tar.gz (1.3 kB)
Collecting post
  Downloading post-2019.4.13.tar.gz (1.3 kB)
Requirement already satisfied: setuptools in c:\program files\python37\lib\site-packages (from request) (39.0.1)
Collecting query_string
  Downloading query-string-2019.4.13.tar.gz (1.6 kB)
Collecting public
  Downloading public-2019.4.13.tar.gz (2.3 kB)
Installing collected packages: public, query-string, get, post, request
    Running setup.py install for public ... done
    Running setup.py install for query-string ... done
    Running setup.py install for get ... done
    Running setup.py install for post ... done
    Running setup.py install for request ... done
Successfully installed get-2019.4.13 post-2019.4.13 public-2019.4.13 query-string-2019.4.13 request-2019.4.13
PS C:\WINDOWS\system32> pip install request

Code: Alles auswählen

easy_install --prefix $HOME/EasyBuild easybuild

Code: Alles auswählen

 
pip install --install-option "--prefix=$HOME/EasyBuild" easybuild
The --prefix $HOME/EasyBuild part in these commands allows you to install EasyBuild without admin rights into $HOME/EasyBuild.

Code: Alles auswählen

 pip install --prefix=$HOME/EasyBuild easybuild works too.

Code: Alles auswählen

 curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py 

Code: Alles auswählen

 python get-pip.py

Code: Alles auswählen

sudo apt-get install python3-pandas

Code: Alles auswählen

sudo apt-get install python3-pandas

Code: Alles auswählen

sudo apt-get install python-pandas

Code: Alles auswählen

Unable to start the Python language server.
Make sure to install pyls 0.19 or newer by running:
python -m pip install 'python-language-server[all]'

Code: Alles auswählen

python3-setuptools

apt-get install python3-setuptools python3-dev gcc g++

pip3 install csv

Code: Alles auswählen

import csv
import requests
from bs4 import BeautifulSoup

url='http://www.mapsofindia.com/districts-india/'
response=requests.get(url)
html=response.content

soup=BeautifulSoup(html,'html.parser')
table=soup.find('table', attrs={'class':'tableizer-table'})
list_of_rows=[]
for row in table.findAll('tr')[1:]:
    list_of_cells=[]
    for cell in row.findAll('td'):
        list_of_cells.append(cell.text)
    list_of_rows.append(list_of_cells)
utfile=open('./immates.csv','w')
writer=csv.writer(outfile)
writer.writerow(["SNo", "States", "Dist", "Population"])
writer.writerows(list_of_rows)

Code: Alles auswählen

Traceback (most recent call last):
  File "/home/martin/.atom/python/examples/bs_mapsofindia.py", line 18, in <module>
    writer=csv.writer(outfile)
NameError: name 'outfile' is not defined
[Finished in 2.835s]

Code: Alles auswählen

#!/usr/bin/env python3
import csv

import requests
from bs4 import BeautifulSoup


def main():
    response = requests.get("http://www.mapsofindia.com/districts-india/")
    response.raise_for_status()
    soup = BeautifulSoup(response.content, "html.parser")
    rows = [
        [cell_node.text for cell_node in row_node("td")]
        for row_node in soup.find("table", "tableizer-table")("tr")[1:]
    ]
    with open("./immates.csv", "w", encoding="utf-8") as utfile:
        writer = csv.writer(outfile)
        writer.writerow(["SNo", "States", "Dist", "Population"])
        writer.writerows(rows)


if __name__ == "__main__":
    main()

Code: Alles auswählen

Version: 1.43.2
Commit: 0ba0ca52957102ca3527cf479571617f0de6ed50
Date: 2020-03-24T21:03:16.125Z
Electron: 7.1.11
Chrome: 78.0.3904.130
Node.js: 12.8.1
V8: 7.8.279.23-electron.0
OS: Linux x64 4.19.0-6-amd64

Code: Alles auswählen

python3 -m pip install --user --upgrade pip

Code: Alles auswählen

python3 -m pip --version
pip 9.0.1 from $HOME/.local/lib/python3.6/site-packages (python 3.6)
Installing virtualenv

Code: Alles auswählen

python3 -m pip install --user virtualenv