Implement ListDataset to iterate over files
See original GitHub issueThis issue is the part of #85
Design
ListDataset
should be very simple. It can iterate over files, apply transforms, and return tensors.
from torch.utils.data import Dataset
import torch
class ListDataset(Dataset):
def __init__(self, filenames_list, transforms, f_imread):
...
def __getitem__(self, idx) -> torch.Tensor:
...
Please, don’t forget to support different types of transforms. Now we use transforms from albumentations
and default torchvision
transforms, you can check it in oml.datasets.base.BaseDataset
.
Transforms are responsible for augmentations of the original images, resizing and normalizing.
Test
Iterate over the files in mock dataset with dataloader and check shapes of batches.
Mock dataset can be found in
from oml.const import MOCK_DATASET_PATH
print(MOCK_DATASET_PATH / 'images')
If you have no mock dataset locally, use make download_mock_dataset
from the root of the project.
Issue Analytics
- State:
- Created a year ago
- Comments:5 (3 by maintainers)
Top Results From Across the Web
How to iterate over files in directory using Python?
This method is used to get an iterator of os.DirEntry objects corresponding to the entries in the directory given by specified path. Example:....
Read more >Iterating through files in a folder - python - Stack Overflow
f is filename but you use it as DataFrame - f['BNF NAME'] . You need df = pd.read_csv(...) and then you can use...
Read more >Read All Files in a Directory - RPubs
This simple loop allows you to read all of the excel files and stack them together to create one final dataset. All files...
Read more >Implement script for extracting features · Issue #85 - GitHub
ListDataset should be very simple. It can iterate over files, apply transforms, and return tensors. from torch.utils.data import Dataset ...
Read more >Create lists of data—ArcMap | Documentation
ArcPy list functions can be used to iterate over a single directory or workspace, but in some cases, iterating through other subfolders and...
Read more >Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start FreeTop Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Top GitHub Comments
@AlekseySh
Thank you for the approval and merge!
I saw on email that I forgot to add ListDataset to documentation. I fixed it in this pull request https://github.com/OML-Team/open-metric-learning/pull/224
https://github.com/OML-Team/open-metric-learning/pull/210