Python automation script for data collection

[noobpythonier]

Hello,
I'm noob in Python and i'd like some help in completing a automation script that collects data from the responses that I get from a series of http requests.
This is a POST request that returns
{value2:00001,value3:some number,value4:some number}
and i want to collect that text and add every time that code is running when gets a response in the line below of .txt file (or export it in csv)
The key part is that of
data = '{value2:00001,value3:null}'
I need some loop to change value1: e.g 00001-00005 and finally to get a file like

{value2:00001,value3:some number,value4:some number}
{value2:00002,value3:some number,value4:some number}
{value2:00003,value3:some number,value4:some number}
{value2:00004,value3:some number,value4:some number}
{value2:00005,value3:some number,value4:some number}

I've tried this python script but no luck

Code: Alles auswählen

import requests

cookies = {
    'value1': 'allow',
}

headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; rv:85.0) Gecko/20100101 Firefox/85.0',
    'Accept': 'application/json',
    'Accept-Language': 'en',
    'Referer': 'https://xyz.com/',
    'Content-Type': 'application/json; charset=utf-8',
    'Authorization': 'some key',
    'X-Requested-With': 'XMLHttpRequest',
    'Origin': 'https://xyz.com',
    'Connection': 'keep-alive',
}

data = '{value2:00001,value3:null}'

response = requests.post('https://xyz.com/api/dervice/get', headers=headers, cookies=cookies, data=data, verify=False)

x = range(00001, 00005)
for var in x:
    print("{value2:"+str(var)+",value3:null}")

with open("response.txt", "w") as f:
    f.writelines(response.text)

Can anyone help?
Thanks!

[__blackjack__]

@noobpythonier: Next time you should also post the error you get. Literal numbers that start with a zero and are not just the zero itself need to have a character denoting the base of the number. E.g. 0b1001 for a binary number, 0x0801 for a number in hexadecimal notation, or 0o664 for octal numbers.

The leading zeroes don't make sense anyway because they don't change the value when the JSON is parsed by the server.

JSON: Your JSON isn't actually JSON because the keys are not quoted. The server may or may not tolerate that. But you shouldn't build some JSON with string operations in the first place. `requests` knows how to handle JSON as data with the `json` keyword argument and the `json()` method on response objects.

You should test the response status instead of assuming everything went fine and the response contains the requested data.

`x` isn't a goot name for a `range` instance and `var` is the name of a built-in function that gets shadowed.

If you want numbers up to and including 5, then the second argument to `range()` must be 6.

When opening a text file there should be an explicit encoding given.

The code may look like this so far:

Code: Alles auswählen

#!/usr/bin/env python3
import json

import requests

HEADERS = {
    "User-Agent": (
        "Mozilla/5.0 (Windows NT 6.1; rv:85.0) Gecko/20100101 Firefox/85.0"
    ),
    "Accept-Language": "en",
    "Referer": "https://xyz.com/",
    "Authorization": "some key",
    "X-Requested-With": "XMLHttpRequest",
    "Origin": "https://xyz.com",
    "Connection": "keep-alive",
}


def main():
    response = requests.post(
        "https://xyz.com/api/device/get",
        headers=HEADERS,
        cookies={"value1": "allow"},
        json={"value2": 1, "value3": None},
        verify=False,
    )
    response.raise_for_status()

    for value in range(1, 6):
        print({"value2": value, "value3": None})

    with open("response.jsonl", "w", encoding="utf-8") as file:
        file.write(json.dumps(response.json()) + "\n")


if __name__ == "__main__":
    main()

If you want to send several requests with the same headers and cookie data, you should use a `requests.Session`. It's a context manager, so use the ``with`` statement.

The ”repackaging” of the JSON data when writing to the file may seem unnecessary but that ensures it is indeed JSON Lines format with the JSON of the response spanning exactly one line in the file. Right now there is just one JSON document in the file, but this gets important when you extend this to several documents.

[noobpythonier]

@von __blackjack__ thank you so much for you help and for your time.
Unfortunately, is not about mistakes. The previous code as this doesn't do what i want.
It saves in both cases a txt (in your case a json) with only one response and not 5 responses, as it should.
Although, it prints 5 lines of request it doesn't respond 5 times but only one.
I guess it doesnt loop for the next numbers in the sequence.
Also "value" in "json={"value2": 1, "MASTER": None} must be somekind of variable rather than 1?
Any suggestions?

[__blackjack__]

@noobpythonier: Yes write the code the way you want it to operate. If you want 5 requests be made and save those 5 responses then write code that doesn't make just one request. And if those 5 requests need a different value in the JSON sent with each request, then don't write code that sends a fixed value.

That's all very basic stuff. You can learn about that in the tutorial in the Python documentation for instance.