question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Implement ListDataset to iterate over files

See original GitHub issue

This issue is the part of #85

Design

ListDataset should be very simple. It can iterate over files, apply transforms, and return tensors.

from torch.utils.data import Dataset
import torch

class ListDataset(Dataset):
    def __init__(self, filenames_list, transforms, f_imread):
        ...

    def __getitem__(self, idx) -> torch.Tensor:
        ...

Please, don’t forget to support different types of transforms. Now we use transforms from albumentations and default torchvision transforms, you can check it in oml.datasets.base.BaseDataset.

Transforms are responsible for augmentations of the original images, resizing and normalizing.

Test

Iterate over the files in mock dataset with dataloader and check shapes of batches.

Mock dataset can be found in

from oml.const import MOCK_DATASET_PATH
print(MOCK_DATASET_PATH / 'images')

If you have no mock dataset locally, use make download_mock_dataset from the root of the project.

Issue Analytics

  • State:closed
  • Created a year ago
  • Comments:5 (3 by maintainers)

github_iconTop GitHub Comments

1reaction
churnikovcommented, Nov 12, 2022

@AlekseySh

Thank you for the approval and merge!

I saw on email that I forgot to add ListDataset to documentation. I fixed it in this pull request https://github.com/OML-Team/open-metric-learning/pull/224

Read more comments on GitHub >

github_iconTop Results From Across the Web

How to iterate over files in directory using Python?
This method is used to get an iterator of os.DirEntry objects corresponding to the entries in the directory given by specified path. Example:....
Read more >
Iterating through files in a folder - python - Stack Overflow
f is filename but you use it as DataFrame - f['BNF NAME'] . You need df = pd.read_csv(...) and then you can use...
Read more >
Read All Files in a Directory - RPubs
This simple loop allows you to read all of the excel files and stack them together to create one final dataset. All files...
Read more >
Implement script for extracting features · Issue #85 - GitHub
ListDataset should be very simple. It can iterate over files, apply transforms, and return tensors. from torch.utils.data import Dataset ...
Read more >
Create lists of data—ArcMap | Documentation
ArcPy list functions can be used to iterate over a single directory or workspace, but in some cases, iterating through other subfolders and...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found