Spark in me

2021-02-12 09:17:45 Multiple pinned posts feature is awesome, please suggest which posts to pin (a link)? Открыть/Комментировать

2021-02-12 09:16:58 Spark in me pinned «Torch Dataloader With Workers Leaking RAM Everyone has faced this issue for HUGE datasets. Is is just because of python itself. If you faced it - you know what I am talking about. I do not claim this to be a definitive solution, but it worked for me. import…» Открыть/Комментировать

2021-02-12 09:16:08 Spark in me pinned «2020 DS / ML Digest 13 Highlights: - Silero models now has an experimental Ukrainian model - CV inference 101 - High-Resolution 3D Human Digitization - Background Features in Google Meet - How to Build an Open-Domain Question Answering System? - A case…» Открыть/Комментировать

2021-02-12 09:15:46 Spark in me pinned «Trying Out New Ampere GPUs and MIG Our hands on experience with new Ampere GPUs - 3090 and A100 (with multi instance GPU) https://habr.com/ru/post/531436/ Please like / share / repost! #hardware #deep_learning» Открыть/Комментировать

2021-02-12 09:15:28 Spark in me pinned «Which Inference Framework Do You Use?» Открыть/Комментировать

2021-02-12 09:14:13 Spark in me pinned «2021 DS / ML Digest 01 Highlights: - Rethinking evaluation in ASR - DALL·E: Creating Images from Text - Deformable Neural Radiance Fields - Analysis of 100 Weeks of Curated AI News Please like / share / repost! https://spark-in.me/post/2021_ds_ml_digest_01…» Открыть/Комментировать

2021-02-12 09:13:39 Spark in me pinned «Итоги 2020 (RU) - https://spark-in.me/post/2020» Открыть/Комментировать

2021-02-12 09:11:18

A small addition to the above post. I did not mention storinators (which are just configurable servers in a custom case ofc).

The cheapest storinator with 16 drive bays and 10 gig ethernet goes for US$3.5k+ wo drives.

So DIY with 10+ drives is 5x cheaper even with 10 gig network / ECC RAM.

If you need 60 drives then there are no options though. Открыть/Комментировать

2021-02-11 17:51:56 ....

A solution? A set of OSS components and off-the-shelf available tools:

- Any MB with at least one or two 10 Gbit/s ethernet ports (the most expensive part) and at least 8-10 SATA slots (PCIE risers can add 2-4 more ports per riser). There are several older mATX SuperMicro boards and a lot of newer overpriced "gaming" boards in this segment

- The cheapest 10 Gbit/s switch or some used "large" switch if you know where / how to buy one

- Any off-the-shelf Fractal Design computer case with 8-10 HDD bays (do not forget 2 "free" 5 inch bays that can be used for 3.5' drives!) or similar (Fractal Design are a bit expensive, but build quality is awesome)

- Any suitable RAM / CPU (for SuperMicro you will have to buy Xeon and ECC RAM)

- Linux as OS (any flavour you like), mdadm for RAID arrays, samba for local sharing

- Just mount your drives locally via fstab (just an example):

//192.168.2.5/share /mnt/share/ cifs username=YOUR_USER,password=YOUR_PASS,iocharset=utf8,uid=YOUR_UID,gid=YOUR_GID,dir_mode=0775,file_mode=0775

- Use any other OSS Linux software, change it as you wish, use any drives you want!

- You can have mdadm, zfs, luks, lvm - whatever you want in any combination;

If you just count the bare minimum cost w/o drives, probably you can get away with US$600-700 per 8 - 16 drives

#hardware Открыть/Комментировать

2021-02-11 17:24:26 Local Network Storage Solution - Hacked

When you do a lot of experiments, depending on your situation a local network shared storage (w or w/o SSD cache) and / or smaller local SSD / NVME usually can cover your needs.

If you google something like "NAS", you will find ridiculously overpriced / underpowered devices, both in B2C and enterprise segments.

All jokes aside, you may find "small" devices for 4 disks starting from US$500-700 with laughable specs (i.e. no 10 Gbit/s ethernet), but with a lot of old proprietary bloatware (i.e. some clunky GUIs not updated for years).

In the "serious" segment - a rack-mounted 12-drive device may cost you north of US$7-10k (luckily with 10 Gbit/s ethernet) . All these prices are without drives. This is totally insane and ridiculous!

.... Открыть/Комментировать