find repetitions in text file and delete repeated lines

[Mehr21]

Hello

I am new to python. I have merged two text files and I am trying to write a python script to find repetitions in these 2 files and delete the repeated lines. Can anyone give me some help on that? Thanks.

[bvdet]

Let's assume you have read in the file as a list of lines and assigned it to the identifier lineList. This can be done with file object method readlines(). You can remove the duplicate lines with a for loop and the in operator.

output = []
for item in lineList:
    if item not in output:
        output.append(item)

This list comprehension works in Python 2.3:

[s for s in lineList if s not in locals()['_[1]'].__self__]

If order is not important, you can use set() or dict(). Example using dict():

>>> lineList = ["A", "B", "C", "C"]
>>> dict.fromkeys(lineList).keys()
['A', 'C', 'B']
>>>

Example using set():

>>> lineList = ["A", "B", "C", "C"]
>>> list(set(lineList))
['A', 'C', 'B']
>>>

[Mehr21]

Thank you so much for the your help.