⏾ Concept Map

Information Theory ○𓆪 Entries

↩ 🔵 🔵 🔵

Primary

၊၊||၊|။

Information Theory ○𓆪｜Definition｜1st｜20260513113259-00-⌔

Information theory - Wikipedia

Information theory

Information theory is the mathematical study of the quantification, storage, and communication of a particular type of mathematically defined information. The field was established and formalized by Claude Shannon in the 1940s,1 though early contributions were made in the 1920s through the works of Harry Nyquist and Ralph Hartley.

Information theory was initially formed in the context of telecommunication but soon found a wide range of other applications. It is now at the intersection of mathematics, statistics and computer science, and has applications in diverse fields ranging from electrical engineering and physics to neurobiology.234

As a simple example of the concept, if one flips a fair coin and does not yet know the outcome (heads or tails), then they lack a certain amount of information. After looking at the coin, they gain information about the outcome. For a fair coin, the probability of either heads or tails is 1/2 and the amount of information is expressed as = 1 bit of information.

A key concept in information theory is entropy. In Shannon’s formulation entropy is equal to the lack of information about an event. In the above coin flip example, the entropy in the case where you don’t know the outcome is 1 bit. When you know the outcome after the coin has landed, the entropy is zero because you have gained one bit.

Information theory has been used in a wide range of applications, such as source coding/data compression (e.g. for ZIP files), and channel coding/error detection and correction (e.g. for DSL). Its impact has been crucial to the success of the Voyager missions to deep space,5 the invention of the compact disc, the feasibility of mobile phones and the development of the Internet and artificial intelligence.674 The theory has also found applications in other areas, including statistical inference,8 cryptography, neurobiology,9 perception,10 signal processing,2 linguistics, the evolution11 and function12 of molecular codes (bioinformatics), thermal physics,313 molecular dynamics,14 black holes, quantum computing, information retrieval, intelligence gathering, plagiarism detection,15 pattern recognition, anomaly detection,16 the analysis of music,1718 art creation,19 imaging system design,20 study of outer space,21 the dimensionality of space,22 and epistemology.23

Printed 2026-06-28.

(echo:: @ ᯤ)

Footnotes

  1. Schneider, Thomas D. (2006). “Claude Shannon: Biologist”. IEEE Engineering in Medicine and Biology Magazine. 25 (1): 30–33. doi:10.1109/memb.2006.1578661. ISSN 0739-5175. PMC 1538977. PMID 16485389. ↩

  2. Cruces, Sergio; Martín-Clemente, Rubén; Samek, Wojciech (2019-07-03). “Information Theory Applications in Signal Processing”. Entropy. 21 (7): 653. Bibcode:2019Entrp..21..653C. doi:10.3390/e21070653. ISSN 1099-4300. PMC 7515149. PMID 33267367. ↩ ↩2

  3. Cao, F. J.; Feito, M. (2009-04-10). “Thermodynamics of feedback controlled systems”. Physical Review E. 79 (4) 041118. arXiv:0805.4824. Bibcode:2009PhRvE..79d1118C. doi:10.1103/PhysRevE.79.041118. ISSN 1539-3755. PMID 19518184. ↩ ↩2

  4. Baleanu, D.; Balas, Valentina Emilia; Agarwal, Praveen, eds. (2023). Fractional Order Systems and Applications in Engineering. Advanced Studies in Complex Systems. London, United Kingdom: Academic Press. p. 23. ISBN 978-0-323-90953-2. OCLC 1314337815. ↩ ↩2

  5. Horgan, John (2016-04-27). “Claude Shannon: Tinkerer, Prankster, and Father of Information Theory”. IEEE. Archived from the original on 2024-11-10. Retrieved 2024-11-08. ↩

  6. Shi, Zhongzhi (2011). Advanced Artificial Intelligence. World Scientific Publishing. p. 2. doi:10.1142/7547. ISBN 978-981-4291-34-7. Archived from the original on 2024-11-10. Retrieved 2024-11-09. ↩

  7. Sinha, Sudhi; Al Huraimel, Khaled (2020-10-20). Reimagining Businesses with AI (1 ed.). Wiley. p. 4. doi:10.1002/9781119709183. ISBN 978-1-119-70915-2. ↩

  8. Burnham, K. P.; Anderson, D. R. (2002). Model Selection and Multimodel Inference: A Practical Information-Theoretic Approach (Second ed.). New York: Springer Science. ISBN 978-0-387-95364-9. ↩

  9. F. Rieke; D. Warland; R Ruyter van Steveninck; W Bialek (1997). Spikes: Exploring the Neural Code. The MIT press. ISBN 978-0-262-68108-7. ↩

  10. Delgado-Bonal, Alfonso; Martín-Torres, Javier (2016-11-03). “Human vision is determined based on information theory”. Scientific Reports. 6 (1) 36038. Bibcode:2016NatSR…636038D. doi:10.1038/srep36038. ISSN 2045-2322. PMC 5093619. PMID 27808236. ↩

  11. cf; Huelsenbeck, J. P.; Ronquist, F.; Nielsen, R.; Bollback, J. P. (2001). “Bayesian inference of phylogeny and its impact on evolutionary biology”. Science. 294 (5550): 2310–2314. Bibcode:2001Sci…294.2310H. doi:10.1126/science.1065889. PMID 11743192. S2CID 2138288. ↩

  12. Allikmets, Rando; Wasserman, Wyeth W.; Hutchinson, Amy; Smallwood, Philip; Nathans, Jeremy; Rogan, Peter K. (1998). “Thomas D. Schneider], Michael Dean (1998) Organization of the ABCR gene: analysis of promoter and splice junction sequences”. Gene. 215 (1): 111–122. doi:10.1016/s0378-1119(98)00269-8. PMID 9666097. Archived from the original on 2008-08-21. Retrieved 2010-10-18. ↩

  13. Jaynes, E. T. (1957). “Information Theory and Statistical Mechanics”. Phys. Rev. 106 (4): 620. Bibcode:1957PhRv..106..620J. doi:10.1103/physrev.106.620. S2CID 17870175. Archived from the original on 2011-08-30. Retrieved 2008-03-13. ↩

  14. Talaat, Khaled; Cowen, Benjamin; Anderoglu, Osman (2020-10-05). “Method of information entropy for convergence assessment of molecular dynamics simulations”. Journal of Applied Physics. 128 (13): 135102. Bibcode:2020JAP…128m5102T. doi:10.1063/5.0019078. OSTI 1691442. S2CID 225010720. ↩

  15. Bennett, Charles H.; Li, Ming; Ma, Bin (2003). “Chain Letters and Evolutionary Histories”. Scientific American. 288 (6): 76–81. Bibcode:2003SciAm.288f..76B. doi:10.1038/scientificamerican0603-76. PMID 12764940. Archived from the original on 2007-10-07. Retrieved 2008-03-11. ↩

  16. David R. Anderson (November 1, 2003). “Some background on why people in the empirical sciences may want to better understand the information-theoretic methods” (PDF). Archived from the original (PDF) on July 23, 2011. Retrieved 2010-06-23. ↩

  17. Loy, D. Gareth (2017), “Music, Expectation, and Information Theory”, in Pareyon, Gabriel; Pina-Romero, Silvia; Agustín-Aquino, Octavio A.; Lluis-Puebla, Emilio (eds.), The Musical-Mathematical Mind: Patterns and Transformations, Computational Music Science, Cham: Springer International Publishing, pp. 161–169, doi:10.1007/978-3-319-47337-6_17, ISBN 978-3-319-47337-6, retrieved 2024-09-19 ↩

  18. Rocamora, Martín; Cancela, Pablo; Biscainho, Luiz (2019-04-05). “Information Theory Concepts Applied to the Analysis of Rhythm in Recorded Music with Recurrent Rhythmic Patterns”. Journal of the Audio Engineering Society. 67 (4): 160–173. doi:10.17743/jaes.2019.0003 (inactive 28 January 2026). ↩

  19. Marsden, Alan (2020). “New Prospects for Information Theory in Arts Research”. Leonardo. 53 (3): 274–280. doi:10.1162/leon_a_01860. ISSN 0024-094X. ↩

  20. Pinkard, Henry; Kabuli, Leyla; Markley, Eric; Chien, Tiffany; Jiao, Jiantao; Waller, Laura (2024). “Universal evaluation and design of imaging systems using information estimation”. arXiv:2405.20559 [physics.optics]. ↩

  21. Wing, Simon; Johnson, Jay R. (2019-02-01). “Applications of Information Theory in Solar and Space Physics”. Entropy. 21 (2): 140. Bibcode:2019Entrp..21..140W. doi:10.3390/e21020140. ISSN 1099-4300. PMC 7514618. PMID 33266856. ↩

  22. Kak, Subhash (2020-11-26). “Information theory and dimensionality of space”. Scientific Reports. 10 (1) 20733. doi:10.1038/s41598-020-77855-9. ISSN 2045-2322. PMC 7693271. PMID 33244156. ↩

  23. Harms, William F. (1998). “The Use of Information Theory in Epistemology”. Philosophy of Science. 65 (3): 472–501. doi:10.1086/392657. ISSN 0031-8248. JSTOR 188281. ↩

Link to original

Secondary

• • •