Mailing List Archive

ok, i came up with
if os.path.isfile(path)
following
path = os.path.join(Lpath, f)

and it seems to be ok, no dupplicates or wrong sizes...

thanks

--
https://mail.python.org/mailman/listinfo/python-list

On 9/23/2020 7:24 PM, pascal z via Python-list wrote:
> Please advise if the following is ok (i don't think it is)
>
> #!/usr/bin/env python3
> # -*- coding: utf-8 -*-
>
> import os
>
> csv_contents = ""
> output_file = '/home/user/Documents/csv/output3csv.csv'
> Lpath = '/home/user/Documents/'
>
> csv_contents = "FOLDER PATH;Size in Byte;Size in Kb;Size in Mb;Size in Gb\n"
>
> d_size = 0
> for root, dirs, files in os.walk(Lpath, topdown=False):
> for i in files:
> d_size += os.path.getsize(root + os.sep + i)
> csv_contents += "%s ;%.2f ;%.2f ;%.2f ;%.2f \n" % (root, d_size, d_size/1024, d_size/1048576, d_size/1073741824)
>
> counter = Lpath.count(os.path.sep)
> if counter < 5:
> for f in os.listdir(Lpath):
> path = os.path.join(Lpath, f)
> f_size = 0
> f_size = os.path.getsize(path)
> csv_contents += "%s ;%.2f ;%.2f ;%.2f ;%.2f \n" % (path, f_size, f_size/1024, f_size/1048576, f_size/1073741824)
>
> fp = open(output_file, "w")
> fp.write(csv_contents)
> fp.close()


Read
https://docs.python.org/3/faq/programming.html#what-is-the-most-efficient-way-to-concatenate-many-strings-together
--
Terry Jan Reedy

--
https://mail.python.org/mailman/listinfo/python-list

On Thursday, September 24, 2020 at 4:37:07 PM UTC+2, Terry Reedy wrote:
> On 9/23/2020 7:24 PM, pascal z via Python-list wrote:
> > Please advise if the following is ok (i don't think it is)
> >
> > #!/usr/bin/env python3
> > # -*- coding: utf-8 -*-
> >
> > import os
> >
> > csv_contents = ""
> > output_file = '/home/user/Documents/csv/output3csv.csv'
> > Lpath = '/home/user/Documents/'
> >
> > csv_contents = "FOLDER PATH;Size in Byte;Size in Kb;Size in Mb;Size in Gb\n"
> >
> > d_size = 0
> > for root, dirs, files in os.walk(Lpath, topdown=False):
> > for i in files:
> > d_size += os.path.getsize(root + os.sep + i)
> > csv_contents += "%s ;%.2f ;%.2f ;%.2f ;%.2f \n" % (root, d_size, d_size/1024, d_size/1048576, d_size/1073741824)
> >
> > counter = Lpath.count(os.path.sep)
> > if counter < 5:
> > for f in os.listdir(Lpath):
> > path = os.path.join(Lpath, f)
> > f_size = 0
> > f_size = os.path.getsize(path)
> > csv_contents += "%s ;%.2f ;%.2f ;%.2f ;%.2f \n" % (path, f_size, f_size/1024, f_size/1048576, f_size/1073741824)
> >
> > fp = open(output_file, "w")
> > fp.write(csv_contents)
> > fp.close()
>
>
> Read
> https://docs.python.org/3/faq/programming.html#what-is-the-most-efficient-way-to-concatenate-many-strings-together
> --
> Terry Jan Reedy

Thanks for this tip. I do think it's better to use lists than concatenate into string variable. However, writing a list to a csv file is not something easy. If strings stored into the list have commas and single quotes (like song title's), it messes up the whole csv when it first meets this. Example with arr as list:


import csv
import io

(...)

csv_contents = "%s;%s;%s;%.2f;%.2f;%.2f;%.2f;%s" % (vfolder_path, vfile_name, vfolder_path_full, 0.00, 0.00, 0.00,0.00, "folder")
arr.append([csv_contents])

b = io.BytesIO()
with open(CSV_FILE,'w', newline ='\n') as f:
#write = csv.writer(f,delimiter=';')
#write = csv.writer(f,quotechar='\'', quoting=csv.QUOTE_NONNUMERIC,delimiter=',')
write = csv.writer(f,b)
for row in arr:
write.writerows(row)

(...)

string samples: ;'Forgotten Dreams' Mix.mp3;'Awakening of a Dream' Ambient Mix.mp3;Best of Trip-Hop & Downtempo & Hip-Hop Instrumental.mp3;2-Hour _ Space Trance.mp3

for the titles above, the easiest thing to write csv for me is


(...)
csv_contents += "%s;%s;%s;%.2f;%.2f;%.2f;%.2f;%s" % (vfolder_path, vfile_name, vfolder_path_full, 0.00, 0.00, 0.00,0.00, "folder"

with open(CSV_FILE,'w') as f:
f.write(csv_contents)


csv_contents can be very large and it seems to work. It can concatenate 30k items and it's ok. Also with the above, I have the expected result into each of the 8 rows having the corresponding data. This is not always the case with csv writerows. If it meets a character it can't process, from there everything go into a single cell row. The split on semi colon from doesnt work anymore.

I am not allowed to change the name of the files (it could be used later somewhere else, making side effects...).
--
https://mail.python.org/mailman/listinfo/python-list

On 04Oct2020 02:56, pascal z <barpasc@yahoo.com> wrote:
>On Thursday, September 24, 2020 at 4:37:07 PM UTC+2, Terry Reedy wrote:
>> Read
>> https://docs.python.org/3/faq/programming.html#what-is-the-most-efficient-way-to-concatenate-many-strings-together
>
>Thanks for this tip. I do think it's better to use lists than
>concatenate into string variable. However, writing a list to a csv file
>is not something easy. If strings stored into the list have commas and
>single quotes (like song title's), it messes up the whole csv when it
>first meets this. [...]
>[...]
>csv_contents = "%s;%s;%s;%.2f;%.2f;%.2f;%.2f;%s" % (vfolder_path,
>vfile_name, vfolder_path_full, 0.00, 0.00, 0.00,0.00, "folder")
>arr.append([csv_contents])
>[...]

Is there a reaon you're not using the csv module to write and read CSV
files. It knows how to correctly escape values in a number of common
dialects (the default dialect works well).

By composing CSV files with %-formatting (or with any crude string
cormatting) you the exact syntax issue you're describing. Faced with
user supplied data, these issues become "injection attacks", as
exemplified by this XKCD comics:

https://xkcd.com/327/
https://www.explainxkcd.com/wiki/index.php/Little_Bobby_Tables

The correct approach here is to have a general and _correct_ formatter
for the values, and to not assemble things with simplistic approaches
like %-formatting.

With databases the standard approach for assembling SQL is to provide
template SQL with the values as arguments, and have the db-specific
driver construct SQL for you. And with CSV files the same applies:
import the csv module and use csv.writer() to general the CSV data; you
just hand the writer an array of values (strings, floats, whatever) and
it takes care of using the correct syntax in the file.

Cheers,
Cameron Simpson <cs@cskk.id.au>
--
https://mail.python.org/mailman/listinfo/python-list