storing data with a ORM - a first approach

[dil_bert]

hello dear experts

 

well - this is a python question - but i do not know how to ask- i am a python novice ...

 

i have a dataset consisting of aprox 7500 lines loooking this way. i want to dive into python programming
and want to store this dataset in a database - i want to use the peewee-db-abstraction
 

{'url': 'http://mypage', 'first-name': 'Phil', 'second-name': 'Windley', 'email': 'phil@mail.com'}
{'url': 'http://mypage', 'first-name': 'bert', 'second-name': 'brien', 'email': 'bert@mail.com'}
{'url': 'http://mypage', 'first-name': 'foo', 'second-name': 'bar', 'email': 'foobar@mail.org'}

i tend top write format that way:-

import peewee
import MySQLdb

con = MySQLdb.connect(user="root", passwd="admin")
cur = con.cursor()
cur.execute('CREATE DATABASE IF NOT EXISTS erp_master')

db = peewee.MySQLDatabase('erp_master', user='root',passwd='admin')

class Bank(peewee.Model):
    bank_id     = peewee.PrimaryKeyField()
    bank_name   = peewee.CharField()
    Account_no  = peewee.IntegerField()
    ifc_code    = peewee.IntegerField()
    swift_code  = peewee.IntegerField(null = True)

    class Meta:
        database = db
db.create_tables([Bank], safe=True)

[requinix]

That's two, now.

 

Okay, so you have some code. What about it?

[dil_bert]

hello dear requinix

 

 

many thanks for the qukc reply. I really am a great fan of this forum here.

 

 

dear requinix - well the thing is - i am a python newbie. I want to dive into the techniques of parsing and storing data.

here i have a solution with the json (java script object notation - the data exchange model) :

 

i like this too. Guess that this should work too;

from peewee import *
import json

db = MySQLDatabase('db', user='iam_the_user',passwd='the_user')

class User(Model):
name = TextField()
name2 = TextField()
email_address = TextField()
url = TextField()

class Meta:
database = db # this model uses the db database

User.create_table() #ensure table is created

data = json.load() #your json data file here

for entry in data: #assuming your data is an array of JSON objects
user = User.create(name=entry["name"], name2=entry["name2"],
email_address=entry["email-adress"], url=entry["url"])
user.save()

will try out what fits best,

 

Working with dictionaries or json or strings...

[requinix]

Solution to what? You haven't actually said what question you have or what problem you're trying to solve. Yes, you have code, but what are you trying to get by posting it here? Feedback? We're a PHP site so don't hold your breath, but if you have a concrete issue then you might be in luck.

[dil_bert]

hello dear requinix

 

many thanks for  the quick reply - great to hear from you. 

 

_I want to store aprox 5000 to 6000 lines of the above mentioned data in a mysql db.

 

Sure thing - this is a PHP-site and i have made the (very) best experience with questions regarding PHP .

 

As a Python novice i am not sure which way to go. But i guess that the second is a very very promising.  I will try out the code at the weekend and will compe back and report all the findings.

 

 

great site here - with such wonderful supporters

 

 

Best reards

[gizmola]

You have a file that has lines of data in JSON format.  However it is not a complete JSON structure, so you will need to read it in line by line:

 

 

 

 
with open('file') as f:
    for line in f:
        data = json.loads(line)
        # Now assign values from data to your mysql fields and save row

[dil_bert]

hello dear gizmola

 

many many thanks for the reply  - great to see your advice. i will try out this at the weekend.

 

Well - youre right - i agree   - this is not a full and true JSON formate:

but i guess that we can call it a dictionary.

 

and with that i can work - i can create a way to access a mysql-db...

 

 

You have a file that has lines of data in JSON format.  However it is not a complete JSON structure, so you will need to read it in line by line:

 
with open('file') as f:
    for line in f:
        data = json.loads(line)
        # Now assign values from data to your mysql fields and save row

 

many thanks - i am prettty new to Python - and with some real world - tasks like using

 

python

- urlparse

- re

- peewee - as a abstraction layer to connect to a mysql-db.

 

i want to dive into Python

 

 

again  - many thanks for your support.

 

 

 i come back and report all the findings

 

greetings dilbert

i want to

Edited October 28, 2017 by dil_bert