# \[SOLVED\] Find duplicates and replace them with adjacent value

**URL:** <https://onehack.st/t/solved-find-duplicates-and-replace-them-with-adjacent-value/61724>\
**Category:** Discussion & Solutions\
**Tags:** solved\
**Created:** [November 5, 2019, 7:17am UTC](https://onehack.st/t/solved-find-duplicates-and-replace-them-with-adjacent-value/61724 "2019-11-05T07:17:05Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![Dracaryu](https://onehack.st/user_avatar/onehack.st/dracaryu/32/18984_2.png) [@Dracaryu](https://onehack.st/u/Dracaryu)\
**Post date:** [November 5, 2019, 7:17am UTC](https://onehack.st/t/solved-find-duplicates-and-replace-them-with-adjacent-value/61724/1 "2019-11-05T07:17:06Z")

</div>

Hi everyone,  
I have a problem, and I would like to ask for help.

I have two text files.

The first one contains multiple lines in this format:  
characterstring : hash  
characterstring1 : hash1  
characterstring2: hash2

The second one is in this format:  
hash: plaintext  
hash2 : plaintext2  
hash3 : plaintext3

I am looking for a way to replace in file 1 all the hashes by their plain text versions contained in the file 2.

The result should be a third file in this format:  
characterstring1:plaintext1  
characterstring2:plaintext2  
characterstring3:plaintext3

Can anyone help me to achieve this? If possible with a python script

Thanks

---

<div class="post-metadata">

**Author:** ![TheJoker](https://onehack.st/user_avatar/onehack.st/thejoker/32/22660_2.png) [@TheJoker](https://onehack.st/u/TheJoker)\
**Post date:** [November 5, 2019, 8:58am UTC](https://onehack.st/t/solved-find-duplicates-and-replace-them-with-adjacent-value/61724/2 "2019-11-05T08:58:24Z")

</div>

## There are some sample python script available, if you can develop your own using these will do the work!

> <https://stackoverflow.com/questions/4746190/find-and-replace-within-a-text-file-using-python>

> <https://stackoverflow.com/questions/10249900/python-beginner-faster-way-to-find-and-replace-in-large-file>

> <https://stackoverflow.com/questions/39755291/find-and-replace-texts-in-all-files-from-the-text-file-input-using-python-in-not>

[https://pythontesting.net/python/regex-search-replace-examples/](https://pythontesting.net/python/regex-search-replace-examples/)

## [Python: Find Replace by Regex](http://xahlee.info/python/findreplace_regex.html)

---

<div class="post-metadata">

**Author:** ![Dracaryu](https://onehack.st/user_avatar/onehack.st/dracaryu/32/18984_2.png) [@Dracaryu](https://onehack.st/u/Dracaryu)\
**Post date:** [November 5, 2019, 1:09pm UTC](https://onehack.st/t/solved-find-duplicates-and-replace-them-with-adjacent-value/61724/3 "2019-11-05T13:09:24Z")

</div>

Sadly, I still struggle to achieve what I’m looking for

---

<div class="post-metadata">

**Author:** ![TheJoker](https://onehack.st/user_avatar/onehack.st/thejoker/32/22660_2.png) [@TheJoker](https://onehack.st/u/TheJoker)\
**Post date:** [November 5, 2019, 2:23pm UTC](https://onehack.st/t/solved-find-duplicates-and-replace-them-with-adjacent-value/61724/4 "2019-11-05T14:23:36Z")

</div>

## It’s hard to scan big text files with any tools, coded script will do the work for you.

---

<div class="post-metadata">

**Author:** ![Shunya](https://onehack.st/user_avatar/onehack.st/shunya/32/12797_2.png) [@Shunya](https://onehack.st/u/Shunya)\
**Post date:** [November 5, 2019, 2:55pm UTC](https://onehack.st/t/solved-find-duplicates-and-replace-them-with-adjacent-value/61724/5 "2019-11-05T14:55:19Z")

</div>

Hi @Dracaryu , You can use pandas for that!  
Something like a vlook operation.

Eg: pokeman is a csv file say  
with data in the below

> <https://gist.github.com/armgilles/194bcff35001e7eb53a2a8b441e8b2c6>

**Now Try this**

`pokemon_names.map(pokemon_types)`

` pokemon_names = pd.read_csv("pokemon.csv", usecols= ["Pokemon"], squeeze= True)`

`pokemon_types = pd.read_csv("pokemon.csv", index_col= "Pokemon", squeeze= True).to_dict()`

` pokemon_names.map(pokemon_types).head()`

Does this help you?

---

<div class="post-metadata">

**Author:** ![Dracaryu](https://onehack.st/user_avatar/onehack.st/dracaryu/32/18984_2.png) [@Dracaryu](https://onehack.st/u/Dracaryu)\
**Post date:** [November 5, 2019, 3:06pm UTC](https://onehack.st/t/solved-find-duplicates-and-replace-them-with-adjacent-value/61724/6 "2019-11-05T15:06:57Z")

</div>

I am not used to that, but I will try.  
Thank you for your help.

---

<div class="post-metadata">

**Author:** ![Shunya](https://onehack.st/user_avatar/onehack.st/shunya/32/12797_2.png) [@Shunya](https://onehack.st/u/Shunya)\
**Post date:** [November 5, 2019, 3:07pm UTC](https://onehack.st/t/solved-find-duplicates-and-replace-them-with-adjacent-value/61724/7 "2019-11-05T15:07:33Z")

</div>

How long is the text with the hash?

---

<div class="post-metadata">

**Author:** ![Dracaryu](https://onehack.st/user_avatar/onehack.st/dracaryu/32/18984_2.png) [@Dracaryu](https://onehack.st/u/Dracaryu)\
**Post date:** [November 5, 2019, 3:10pm UTC](https://onehack.st/t/solved-find-duplicates-and-replace-them-with-adjacent-value/61724/8 "2019-11-05T15:10:32Z")

</div>

530k for the first one.  
174k for the second one.

---

<div class="post-metadata">

**Author:** ![Shunya](https://onehack.st/user_avatar/onehack.st/shunya/32/12797_2.png) [@Shunya](https://onehack.st/u/Shunya)\
**Post date:** [November 5, 2019, 3:16pm UTC](https://onehack.st/t/solved-find-duplicates-and-replace-them-with-adjacent-value/61724/9 "2019-11-05T15:16:20Z")

</div>

Oh, you can try pandas, But it might be less efficient.  
Here, I found some more helpful links.

This might actually help you.

> <https://codereview.stackexchange.com/questions/124964/vlookup-in-python>

---

<div class="post-metadata">

**Author:** ![Dracaryu](https://onehack.st/user_avatar/onehack.st/dracaryu/32/18984_2.png) [@Dracaryu](https://onehack.st/u/Dracaryu)\
**Post date:** [November 5, 2019, 3:21pm UTC](https://onehack.st/t/solved-find-duplicates-and-replace-them-with-adjacent-value/61724/10 "2019-11-05T15:21:27Z")

</div>

Thanks. I will check that.