navigation

GTDLBench

With its effective and wide real applications, deep learning attracts great attention nowadays. Many deep learning frameworks emerged to facilitate deep learning research, development and applications. However, few research has been conducted to effectively measure these frameworks. And it still remains a critical task on choosing the optimal framework for specific deep learning models and applications. GTDLBench aims at introducing a hybrid benchmark to systematically measure the performance of these deep learning frameworks as well as the performance of different hardware devices. The performance metrics derived from both deep learning and systems field are introduced to present an easy and holistic approach for deep learning benchmarking.

Metrics

End-to-end Benchmark

Micro Benchmark

Tutorials

Deep Learning Frameworks

Frameworks

Datasets

Deep Learning Models

Please cite the following papers:

  • Benchmarking Deep Learning Frameworks: Design Considerations, Metrics and Beyond
  • A Comparative Measurement Study of Deep Learning as a Service Framework
  • Experimental Characterizations and Analysis of Deep Learning Frameworks
  • @INPROCEEDINGS{GTDLBenchICDCS, 
        author={{Liu}, Ling and {Wu}, Yanzhao and {Wei}, Wenqi and {Cao}, Wenqi and {Sahin}, Semih and {Zhang}, Qi}, 
        booktitle={2018 IEEE 38th International Conference on Distributed Computing Systems (ICDCS)}, 
        title="{Benchmarking Deep Learning Frameworks: Design Considerations, Metrics and Beyond}",
        year={2018},
        pages={1258-1269}, 
        doi={10.1109/ICDCS.2018.00125}, 
        ISSN={2575-8411}, 
        month={July},
    }
    
    @ARTICLE{GTDLBencharTSC,
        author={Y. {Wu} and L. {Liu} and C. {Pu} and W. {Cao} and S. {Sahin} and W. {Wei} and Q. {Zhang}}, 
        journal={IEEE Transactions on Services Computing}, 
        title={A Comparative Measurement Study of Deep Learning as a Service Framework}, 
        year={2019}, 
        volume={}, 
        number={}, 
        pages={1-1}, 
        keywords={Libraries;Parallel processing;Hardware;Training;Runtime;Deep learning;Task analysis;Deep Learning as a Service;Big Data;Deep Neural Networks;Accuracy}, 
        doi={10.1109/TSC.2019.2928551}, 
        ISSN={1939-1374}, 
        month={},
    }
    
    @INPROCEEDINGS{GTDLBenchBigData, 
        author={{Wu}, Yanzhao and and {Cao}, Wenqi and {Sahin}, Semih and {Liu}, Ling}, 
        booktitle={2018 IEEE 38th International Conference on Big Data}, 
        title="{Experimental Characterizations and Analysis of Deep Learning Frameworks}", 
        year={2018},
        month={December},
    }