SeeJaySea.dev

Devops In Small Teams

Mon, 09 Nov 2020 05:00:00 GMT

Modern software products are becoming larger and more complicated every year. In addition to this added complexity, the digital landscape of today requires faster development and release cycles. Companies like Google, Amazon, Facebook, and Netflix are characterized by their fast-paced releases and quick response times. The solution to the problem often lies within the integration and streamlining of software development and IT operations; this integration has become known as DevOps. With DevOps becoming mainstream and adopted by tech giants, does a DevOps pipeline make sense for all companies? Does the time saved and fast releases make financial sense to offset the initial obstacles and startup costs?

<h2>Why a Small Team Would Want DevOps</h2> Software development within the largest tech companies, colloquially known as FAANG companies<a href="#Source2" data-type="internal" data-id="#Source2">[2]</a>, can sometimes seem like a race to first release. The race to release has led many large companies to invest largely in sophisticated DevOps infrastructures. DevOps, a generic term used to describe the integration of software development and IT operations, has very quickly become the standard for large scale software development. 88% of 1425 organizations had already or planned to adopt DevOps practices before 2019<a href="#Source7" data-type="internal" data-id="#Source7">[7]</a>. While the benefits of the DevOps practice are very often touted, is this model practical and sustainable for smaller organizations? Management personnel could easily see Facebook’s claim that they are capable of deploying a continuous stream of hundreds and up to thousands of updates per day<a href="#Source5">[5]</a> and assume their business could benefit from at least some of the methods being employed by Facebook. The concept of unified development and operations in theory is perfect for any size team building software. The main idea of DevOps is to create an environment of collaboration between development and operation teams<a href="#Source8">[8]</a> avoiding and mitigating scenarios of the development and operation teams blaming one another for any failures during deployment of a software product. However, implementing new development technologies and technologies is never hassle-free.

<h2>Complications with Existing Management</h2>

Without buy-in on DevOps methodology from all of senior management, the attempt to begin operating with DevOps in mind could run into multiple roadblocks.<a href="#Source3">[3]</a> As seen in the initial findings from Jones et al, <a href="#Source3">[3]</a> an existing managerial structure can easily undermine a Software Development Manager (SDM) and their attempts to establish a DevOps plan. An example can be seen in the UK Small to Medium Enterprise (SME) Jones et al performed their qualitative study on. In the initial findings, the existing managerial structure had the SDM and the IT Operations team both reporting to the Head of Group Operations. Because of this structure, a senior system administrator was able to circumvent the SDM while the team was trying to implement a DevOps workflow. The solution proposed by Jones et al to close the existing “IT operations loophole” is simple, the Software Development Team and IT Operations Team should both report to the SDM, who then reports to the Head of Group Operations. However, this solution is not feasible to implement without the full investment of senior leadership into the transition to a DevOps approach.

<h2>Complications with People and Money</h2>

Within a smaller engineering team, it can be more common for individuals to fill multiple roles. A team within a company that isn’t solely focused on software development can easily have a team size of under 10 engineers, similar to the size of some startups.<a href="#Source6">[6]</a> This lack of personnel available could add unneeded stress to the average worker within a small company during the integration of a DevOps workflow. Employees in the development and operations teams would still be required to perform their normal daily tasks, in addition to attempting to build up the systems to support DevOps. Companies looking to implement DevOps principles could also fall into a similar situation seen in the 2016 UK SME study conducted by Jones et al. Within this study, the company being observed was attempting to implement DevOps for the development of a new system as well as maintaining a legacy system that was critical to business operations.<a href="#Source3">[3]</a> Small engineering teams will see similar situations occur if the implementation of DevOps will be paired with the development of a new system. These smaller teams may lack personnel and/or funding to accomplish everything within a timeframe seen as economically viable by upper managers.

<h2>Initial Costs</h2>

Because of its growth in popularity, the tools used for DevOps have become their own submarket within the software development industry. Like many other aspects within software development, there is a myriad of tools available to allow companies to attempt to integrate a DevOps workflow at various prices as well. The integration of a new DevOps workflow can theoretically be completed with tools that do not require licensing fees and operate under open-source licensing. However, the true cost of open-source software solutions is not normally a tangible dollar figure that can be provided to the company’s accounting department. Often open-source solutions require bootstrapping the entire system from the ground up, whereas a more expensive enterprise-grade solution may come with the advantage of a more simple startup and integration or customer support to aid the team through the integration steps. A pricing model that is becoming more common is software products offering low cost or free low tier plans with increasing costs dependent on the use case. The advantage of this newer pricing model is that a small team can begin the integration of these tools without having to justify the cost of DevOps tools to upper management until a tangible benefit of the tool’s integration can be seen and more easily justify the price of an enterprise license.

<h2>Security Concern</h2>

When conducting research into DevOps, it is very easy to see all the benefits of getting a powerful and efficient continuous integration / continuous delivery (CI/CD) system running. It would be plausible for a manager to see the constant delivery systems from companies like Facebook reporting up to thousands of releases within a single day<a href="#Source5">[5]</a> and assume if their team could reach that level of efficiency, most of their engineering problems could be solved. The importance of integrating security within the DevOps process should not be forgotten. In fact, many software developers have stressed this importance and created vernacular in accordance with Secure DevOps, commonly referred to as SecDevOps or DevSecOps.<a href="#Source4">[4]</a> The continuous deployment numbers of Facebook are often quoted in order to tout the efficiency of a well-engineered DevOps pipeline. What is often overlooked is that any software changes related to end-user privacy are handled differently and the deployment rate is limited.<a href="#Source1">[1]</a> While some DevOps practices such as automated monitoring, testing, and deployment may actually be beneficial to a system’s security.<a href="#Source4">[4]</a> The implementation of DevOps practices strongly promotes the idea of collaboration amongst the Development and Operations teams. Rahman & Williams<a href="#Source4">[4]</a> observed that unrestricted collaboration could lead to individuals “inappropriately gaining access to system resources and accidentally or deliberately changing system properties” this change could very well have a negative impact on the overall system security. Rahman & Williams<a href="#Source4">[4]</a> observed that established DevOps organizations had security awareness practices and active involvement from security teams for collaboration on DevOps. It should be imperative that the adoption of DevOps is done with the collaboration of Information Assurance (IA) personnel in order to protect the end-user’s security.

<h2>Conclusion</h2>

DevOps/SecDevOps is definitely the way large scale tech companies will be developing and releasing products for the foreseeable future. DevOps within FAANG companies can allow them to gain an edge in the modern software arms race, providing companies the capability to premiere a new feature to the end-user before their competitors. With the advantages DevOps brings to the largest tech companies, it is understandable that a smaller development team may want to adopt their own form DevOps. The complications that a small development team may run into during the start-up of a DevOps workflow are likely similar to the challenges faced within the largest companies. However, companies on the FAANG scale have the advantages of financial backing and manpower to work overcome obstacles that arise. The adoption of DevOps is not impossible for a small team, and while the obstacles that will be encountered are not impossible to overcome, they should be considered and known before the integration of DevOps begins. Once a team is allowed the freedom to change how release and deployment of software will be handled as well as getting support from most, if not all, of the Development and IT Operations teams the path to DevOps will have many obstacles unique to smaller teams lifted.

<h3>References</h3>

<ol><li><a name="Source1" href="https://ieeexplore.ieee.org/document/6449236" target="_blank" rel="noopener noreferrer">Feitelson, D., Frachtenberg, E., & Beck, K. L. (2013, February 4). Development and Deployment at Facebook. https://ieeexplore.ieee.org/document/6449236&nbsp</a> </li><li><a name="Source2" href="https://www.investopedia.com/terms/f/faang-stocks.asp">Fernando, J. (2020, August 28). FAANG Stock. Investopedia. https://www.investopedia.com/terms/f/faang-stocks.asp.</a> </li><li><a name="Source3" href="https://doi.org/10.1145/2945408.2945410" target="_blank" rel="noopener noreferrer">Jones, S., Noppen, J., & Lettice, F. (2016). Management challenges for DevOps adoption within UK SMEs. Proceedings of the 2nd International Workshop on Quality-Aware DevOps – QUDOS 2016. https://doi.org/10.1145/2945408.2945410</a> </li><li><a name="Source4" href="https://doi.org/10.1145/2896941.2896946" target="_blank" rel="noopener noreferrer">Rahman, A. A. U., & Williams, L. (2016). Software security in DevOps. Proceedings of the International Workshop on Continuous Software Evolution and Delivery – CSED ’16. https://doi.org/10.1145/2896941.2896946</a> </li><li><a name="Source5" href="https://doi.org/10.1145/2889160.2889223" target="_blank" rel="noopener noreferrer">Savor, T., Douglas, M., Gentili, M., Williams, L., Beck, K., & Stumm, M. (2016). Continuous deployment at Facebook and OANDA. Proceedings of the 38th International Conference on Software Engineering Companion – ICSE ’16. https://doi.org/10.1145/2889160.2889223</a> </li><li><a name="Source6" href="https://doi.org/10.1109/softstart.2017.2" target="_blank" rel="noopener noreferrer">Souza, R., Malta, K., & Almeida, E. S. D. (2017). Software Engineering in Startups: A Single Embedded Case Study. 2017 IEEE/ACM 1st International Workshop on Software Engineering for Startups (SoftStart). https://doi.org/10.1109/softstart.2017.2</a> </li><li><a name="Source7" href="https://infotech.report/Resources/Whitepapers/dc26fd4e-c270-40c4-ab3e-35bfe82752ef_devops-winning-in-application-economy-2.pdf" target="_blank" rel="noopener noreferrer">Technologies, C. A. (2014). DevOps: The Worst-Kept Secret to Winning in the Application Economy. Https://Infotech.report/. https://infotech.report/Resources/Whitepapers/dc26fd4e-c270-40c4-ab3e-35bfe82752ef_devops-winning-in-application-economy-2.pdf.</a> </li><li><a name="Source8" href="https://doi.org/10.1109/icsess.2015.7339039" target="_blank" rel="noopener noreferrer">Wahaballa, A., Wahballa, O., Abdellatief, M., Xiong, H., & Qin, Z. (2015). Toward unified DevOps model. 2015 6th IEEE International Conference on Software Engineering and Service Science (ICSESS). https://doi.org/10.1109/icsess.2015.7339039</a> </li></ol>

Data Mining and Modern Advertising

Thu, 04 Mar 2021 05:00:00 GMT

Some of the largest companies in the new digital age are built upon no cost services. The Google search engine and social media sites like Facebook and YouTube began to dominate the internet in the age known as Web 2.010. The profits of these companies are built upon an economy of human attention and advertisements. Such platforms are incentivized to retain users’ attention and the most successful companies can achieve this through subtle psychological manipulation. While the systems being developed to facilitate this economy of human attention are impressive, the software professionals working on the current systems that allow for this manipulation of users are no longer complying with the ACM/IEEE-CS Software Engineering Code of Ethics. These systems cannot exist without being in violation of principal one: “Software engineers shall act consistently with the public interest”5. Though these systems are currently being used to encourage longer session times and keep the end-user’s attention for longer, research is being conducted on the application of these systems to contribute good to the general public, not only to company profits.

Data Mining

Data Mining and Modern Advertising The term “Data Mining” as it is known today was introduced in the 1990s13. The concept of Data Mining has become much more prevalent in “Web 2.0”. Web 2.0, coined in the early 2000s to refer to a turning point in the post-dot-com bubble internet10, is the age where the internet became increasingly filled with user-generated content. This increase in user-generated content also provided internet-based companies incredibly large amounts of data. In 2018, Forbes estimated that 2.5 quintillion bytes of data were being produced every day9 and YouTube itself reports that 500+ hours of video content is uploaded to their platform every minute 18.

What is Data Science?

To leverage the amount of available data, extraction only processing of the data is not enough 13. In order to take full advantage of the data available, summation and interpretation is necessary. The practice of building models to facilitate this analysis is encapsulated under the field of “Data Science”. The process of extraction, summation, and interpretation falls largely into two main models, the “predictive model” and the “descriptive model” 2. The predictive model is used to predict an outcome from a given dataset based upon target variables, and the descriptive model is used to gain a “better understanding of the data, without any single specific target variable” 2. Simply put, the practice of data science is the process to, as Jeanette Wing says, “extract value from data”17.

Data Mining in Social Media

Throughout the modern internet, applications of the adage, “if you are not paying for the product, you are the product” can be found. For a large-scale tech company like Facebook, whose service is provided at no charge to the average user, one can believably question how a company at that size can be profitable. Facebook is a corporation with the technology in place to leverage their large user base for the benefit of businesses using their Pixel system4.

The Facebook Pixel

The Facebook Pixel is a system that allows a company to track users interactions on a website, conversions (sales) from advertisements placed onto one of Facebook’s platforms, as well as facilitating retargeting of users that have either shown interest in a specific product or a related product4. The Facebook Pixel system is based upon the practice of placing “cookies” into the end-user’s browser4. These cookie-based tracking systems have recently come under fire because they allow the large tech platforms the capability to perform both deterministic and predictive analysis on the end-user. The deterministic analysis allows these systems to determine a user’s demographics (e.g. age or race) in addition to their interests11. The information for the deterministic analysis is then available to businesses to see who their user base consists of. After this analysis is complete, these systems are capable of directly targeting end-users to “create more relevant advertisements” 4.

Algorithmically Building an Echo Chamber

These systems are highly beneficial to companies and advertisers, and various social media platforms are all simultaneously fighting for the attention of their users. Companies like YouTube, Facebook, Reddit, and Netflix each want to maximize engagement time with their users. One of the easiest ways to increase the engagement time with the end-user is to have a continual feed of information relevant to the specific user11. This feed will be built from the predictive analysis generated for the express purpose to retain a user’s interest. Unless a user specifically searches for information with differing viewpoints, this process slowly builds an “echo chamber” where the user’s beliefs are only confirmed and not challenged directly15. The presence of digital echo chambers may be a contributing cause of the increasing polarization between political ideologies12. The following figures are taken from a 2017 Pew Research report

<caption>Division between median political views in 1994</caption>

<caption>Division between median political views in 2011</caption>

<caption>Division between median political views in 2017</caption>

Data Mining Aided Suicide Prevention

Data mining does have a use for altruistic means, and Jeannette Wing from Columbia University has started promoting the idea of “Data for Good”17. This concept is based on two main ideas. First, that data science should be used to better people’s lives, and second data should be used in a “good” manner17. An example of data being used in a “good” manner can be seen in the proof of concept conducted by Seah and Shim14. Data science models can be used on individual users of a social media site to provide detection of language associated with suicide and self-harm14. With suicide as the 2nd leading cause of death in the age range of 10 to 34 years old1, the work conducted by Seah and Shim demonstrated that there is a valid use case to use predictive models to indicate users that may be at risk for self-harm14.

Conclusion

Companies currently competing within the market of human attention are currently required to produce products that are good for the company and the shareholders above all. Since these companies are developing systems which compete for the same attention pool, the designers of these systems are slowly creating technology that has the potential for addictive behavior. As the Center for Humane Technology states on their website “As long as social media companies profit from outrage, confusion, addiction, and depression, our well-being and democracy will continue to be at risk”7. The concept of addictive technology has become so prevalent that even the CEO of Twitter Jack Dorsey has acknowledged this idea when being questioned by the U.S. Senate Judiciary Committee6:

I do think, like anything else, these tools can be addictive, and we should be aware of that, acknowledge it, and make sure that we are making our customers aware of better patterns of usage. The more information the better here.

-- <cite>Jack Dorsey</cite>

References

1.<a name="ref-1"></a> Biernesser, C., Sewall, C. J. R., Brent, D., Bear, T., Mair, C., & Trauth, J. (2020). Social media use and deliberate self-harm among youth: A systematized narrative review. Children and Youth Services Review, 116. https://doi.org/https://doi-org.proxy.lib.fsu.edu/10.1016/j.childyouth.2020.105054 2.<a name="ref-2"></a> Boonjing, V., & Pimchangthong, D. (2017). Data Mining for Customers’ Positive Reaction to Advertising in Social Media. Proceedings of the 2017 Federated Conference on Computer Science and Information Systems https://doi.org/10.15439/2017f356 3. <a name="ref-3"></a>Curtin, S. C. (2020, September 11). National Vital Statistics Reports. https://www.cdc.gov/nchs/data/nvsr/nvsr69/NVSR-69-11-508.pdf 4. <a name="ref-4"></a>Facebook. (2020, February). Business Help Center. Facebook Business Help Center. https://www.facebook.com/business/help/742478679120153?id=1205376682832142 5. <a name="ref-5"></a>Gotterbarn, D., Miller, K., & Rogerson, S. (2018, December 19). Software Engineering Code. ACM Ethics. https://ethics.acm.org/code-of-ethics/software-engineering-code/ 6. <a name="ref-6"></a>Hartmans, A. (2020, November 17). Jack Dorsey says social media platforms like Twitter and Facebook can be addictive - Mark Zuckerberg says the research is 'inconclusive'. Business Insider. https://www.businessinsider.com/twitter-jack-dorsey-social-media-can-be-addictive-2020-11 7. <a name="ref-7"></a>Join the movement for Humane Technology. Center for Humane Technology. (2020). https://www.humanetech.com/ 8. <a name="ref-8"></a>Koch, R. (Ed.). (2019, May 9). Cookies, the GDPR, and the ePrivacy Directive. GDPR.eu. https://gdpr.eu/cookies/?cn-reloaded=1 9. <a name="ref-9"></a>Marr, B. (2018, May 21). How Much Data Do We Create Every Day? The Mind-Blowing Stats Everyone Should Read. Forbes. https://www.forbes.com/sites/bernardmarr/2018/05/21/how-much-data-do-we-create-every-day-the-mind-blowing-stats-everyone-should-read/?sh=6bc68e9560ba 10. <a name="ref-10"></a>O'Reilly, T. (2005, November 30). What Is Web 2.0. O'Reilly. https://www.oreilly.com/pub/a/web2/archive/what-is-web-20.html 11. <a name="ref-11"></a>Rhodes, L. (Producer), & Orlowski, J (Director). (2020). The Social Dilemma [Video file]. Retrieved from https://www.netflix.com/title/81254224 12. <a name="ref-12"></a>Pew Research Center. (2017, October 5). The Partisan Divide on Political Values Grows Even Wider. Pew Research Center. https://www.pewresearch.org/politics/2017/10/05/the-partisan-divide-on-political-values-grows-even-wider/. 13. <a name="ref-13"></a>Ramzan, M., & Ahmad, M. (2014). Evolution of data mining: An overview. 2014 Conference on IT in Business, Industry and Government (CSIBIG). https://doi.org/10.1109/csibig.2014.7056947 14. <a name="ref-14"></a>Seah, J. H. K., & Shim, K. J. (2018). Data Mining Approach to the Detection of Suicide in Social Media: A Case Study of Singapore. 2018 IEEE International Conference on Big Data. https://doi.org/https://doi-org.proxy.lib.fsu.edu/10.1109/BigData.2018.8622528 15. <a name="ref-15"></a>Seneca, C. (2020, September 17). How to Break Out of Your Social Media Echo Chamber. Wired. https://www.wired.com/story/facebook-twitter-echo-chamber-confirmation-bias/ 16. <a name="ref-16"></a>Wing, J. M. (2018a). Data for Good: Abstract. Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. https://doi.org/10.1145/3219819.3219942 17. <a name="ref-17"></a>Wing, J. M. (2018b, August 19). Data for Good: Keynote Address. KDD ’18: The 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. Retrieved November 15, 2020, from https://doi.org/10.1145/3219819.3219942 18. <a name="ref-18"></a>YouTube. (2020, September 24). YouTube for Press. blog.youtube. https://blog.youtube/press/

R for Stats and Visualizations

Fri, 30 Apr 2021 04:00:00 GMT

R is an extremely popular environment amongst statisticians, data analysts, and computer science. From the beginnings of a project between two colleagues interested in statistical computing to one of the most popular programming environments, R has evolved to being used not only in academia but in business settings. Even on its initial release, R was adopted quickly by statisticians and engineers without programming knowledge. R is capable of being run in numerous environments and is known for its capabilities of not only performing statistical calculations but also for developing graphics in a flexible manner. R has quickly become a standard in both industry and academia for computation and visualizations. It is also capable of being used for algorithms in Machine Learning and distributed computing systems.

R's History

Two colleagues from the Statistics Department at the University of Auckland in New Zealand, Ross Ihaka and Robert Gentleman, saw a need for a better software environment. Both had interest in statistical computation and desired functionalities like that of Scheme, and they began work on a small Scheme-like interpreter for statistics computations. The consisting of around 1000 lines of C code, the initial interpreter provided a good deal of R's functionality.<a href="#ref-15">15</a> With their familiarity of S, they choice to use an S-like syntax and according to Ihaka, "this decision, more than anything else, has driven the direction that R development has taken."<a href="#ref-15">15</a>

Open Sourcing

In August of 1993, after an initial version of R was implemented, Ihaka and Gentlemen placed some binary copies on Statlib, a system for distributing statistical software, datasets, and information. Several people picked up the binaries and provided feedback to Ihaka and Gentleman.<a href="#ref-15">15</a> Martin Mächler of ETH Zurich persisted that the two of them release R as “free software” and, in June of 1995, Ihaka and Gentlemen released R under the GPL. By 1997, a larger “Core Group” was integrated into the main development of R and the Comprehensive R Archive Network (CRAN) was established.<a href="#ref-8"></a> Since 1997, write access to the R source code has been limited to the R Core Team and bug reports are accepted from the users of R via email.<a href="#ref-13">13</a>

The Language

The R language is a statistical computation environment licensed under the GNU General Public License (GPL) with syntax based on the S language, another high-level statistical computation language.<a href="#ref-9">9</a> The R environment also allows interfacing with procedures written in C, C++, or FORTRAN. The base distribution of R contains functionality for many statistical procedures as well as access to numerous packages through CRAN.<a href="#ref-9">9</a>

Differences Between R and S

While the syntax of R is heavily influenced by S, two fundamental differences exist between R and S as a result of R's Scheme heritage, the use of a garbage collector for memory management and lexical scoping.<a href="ref-15">15</a>

Memory Management

On startup, R allocates a block of memory and uses a garbage collection strategy to manage internal objects. At the end of a session, the current objects within R's memory can be saved.<a href="ref-3">3</a>

Lexical Scoping

Variables within R are lexically scoped.<a href="ref-9">9</a> Because of this, variables can be accessed within the context in which they are defined.

cube <- function(n) {
    sq <- function() n * n
    return n * sq()
}

Environments

R consists of the R language, a run-time environment, and debugger.<a href="ref-15">15</a> R can be run in Read-Eval-Print-Loop (REPL) mode or files can be written and called later. R Programs can be written and supports multiple third-party environments such as, Jupyter Notebooks, RStudio, Apache Spark, and R Tools for Visual Studio (RTVS).

Jupyter Notebooks

Named for the core supported languages Julia, Python, and R, Jupyter Notebooks are an opensource web application that allows the user to create and store documents containing live code, equations, visualizations, and text.<a href="ref-12">12</a> Jupyter Notebooks are also gaining more widespread adoption outside of statistic and computer science curricula due to the proliferation of large data sets.<a href="ref-6">6</a>

Apache Spark

Apache Spark is a cluster computing framework for large-scale data processing which focuses on efficiency and fault tolerance.<a href="ref-7">7</a> SparkR is an R package introduced to allow R to integrate with the Apache Spark Unified Analytic Engine. SparkR also supports functionality for distributed machine learning using the MLlib package.

RStudio

RStudio is an Integrated Development Environment (IDE) for R consisting of a console and an editor supporting direct code execution as well as tools for plotting, history, debugging, and workspace management.<a href="ref-16">16</a>

R Tools for Visual Studio

In Visual Studio 2017, Microsoft introduced the R Tools for Visual Studio (RTVS) giving data analysts access to the tooling for developing R within Visual Studio in addition to Microsoft R Open, Microsoft’s distribution of R and the Microsoft R Server libraries for accelerated computation on datasets that do not fit into a system memory.<a href="ref-5">5</a>

Packages and Graphics

Early in the life cycle of R, Ihaka started experimenting with R’s graphics. Early experimentation involved R’s color models and line textures. R uses a device independent 24-bit model for colors capable of operating in Hexadecimal, X Windows, and S-Compatible modes.<a href="ref-15">15</a> In addition to its multiple color operation modes, R is capable of rendering lines themselves in several ways. Line textures can be defined through common names (e.g., dotted, dashed, etc.).<a href="ref-15">15</a> Line textures can also be defined as a string containing segments lengths. For example, an up/down segment definition of 52 translates to a pen down of 5 points (or pixels) followed by a pen up of 2 points (or pixels).<a href="ref-15">15</a> Like the color modes, R also has an S compatible mode for line rendering allowing the textures to be defined through an index to a set of line types.<a href="ref-15">15</a> R’s functionality can also be extended with several software packages allowing additional functionality for almost anything from Markdown, ODBC integration, or visualization utilities. R packages can add the functionality of static visualizations through packages such as ggplot2 as well as the development of dynamic visualizations through packages like leaflet.<a href="ref-5">5</a> R is known industry-wide for its graphics rendering capabilities.

Popularity

In 2020, the Institute of Electrical and Electronics Engineers (IEEE) has ranked the R Programming Language as the number 6 programming language.<a href="#ref-4">4</a> In addition to this software professionals that responded to the 2020 Stack Overflow Survey of places R as one of the most loved languages available.<a href="#ref-17">17</a> The R language has become so popular with graduate-level students that Max Khun, the Associate Director of Nonclinical Statistics at Pfizer says, “R has become a second language for people coming out of grad school now, and there’s an amazing amount of code being written for it.”<a href="#ref-2">2</a>

Applications

R has found a devoted following of statisticians, engineers, and scientists without computer programming skills, and organizations like Google and Pfizer are using R for many additional workloads.<a href="#ref-2">2</a> In addition to professional workloads, R is very common in academic research and has been used to produce algorithms for text mining<a href="#ref-1">1</a>, machine learning with Support Vector Machines (SVM)<a href="#ref-11">11</a>, and for the design and test of digital logic DNA systems.<a href="#ref-14">14</a> In 2019, de Lima et al developed an algorithm with R to conduct patent analysis in order the identify the stage of technological development of photovoltaic panels. The team's study found an increase in patent deposits resulting in increased availability of lower cost and more efficient panels.<a href="#ref-1">1</a> In 2021, Sanghar et al used R to implement a Support Vector Machine (SVM) in order to predict diabetes. An SVM is a supervised machine learning model allowing for the accomplishment of tasks through machine training. An SVM is most used for classification and linear regression. <a href="#ref-11">11</a> In 2021, Marks et al began looking at a new method of circuit design using synthetic DNA molecules as the substrate.The team extended an existing R package, DNAr, to aid in the construction of DNA strand displacement (DSD) systems to build circuits from synthetic DNA molecules. The aim of this extended package was to make design and simulation of these DSD circuits to be more accessible.<a href="#ref-14">14</a>

Wrapping Up

Though it started as a very small project with two main developers, in the 30 years after its introduction R has gained vast popularity. Almost immediately, statisticians and non-programming engineers began using R for statistical computations and analysis. It quickly rose to favor within academics and has become a standard tool for data analysts and data scientists in the workforce. In addition to being an incredibly robust statistics environment, R can create and customize visualizations to represent the outcomes of these calculations. R has recently become one of the top desired languages amongst professional developers.

References

<a name="ref-1" href="https://www.kaggle.com/michau96/covid-19-continents-in-relation-to-time/output" target=blank>Bogacz, M. (2021, March 1). COVID-19: Continents in relation to time. Kaggle. https://www.kaggle.com/michau96/covid-19-continents-in-relation-to-time/output</a>
<a name="ref-2" >de Lima, A., Argenta, A., Zattar, I., & Kleina, M. (2019). Applying Text Mining to Identify Photovoltaic Technologies. IEEE Latin America Transactions, 17(05), 727–733. https://doi.org/10.1109/tla.2019.8891940</a>
<a name="ref-3" >Hildebrandt, K., Panse, F., Wilcke, N., & Ritter, N. (2020). Large-Scale Data Pollution with Apache Spark. IEEE Transactions on Big Data, 6(2), 396–411. https://doi.org/10.1109/tbdata.2016.2637378</a>
<a name="ref-4" >Hornik, K. (1997, April 23). ANNOUNCE: CRAN. https://stat.ethz.ch/pipermail/r-announce/1997/000001.html</a>
<a name="ref-5" >Hornik, K. (2020, February). R FAQ. https://cran.r-project.org/doc/FAQ/R-FAQ.html</a>
<a name="ref-6" >IEEE. (2020). IEEE Spectrum. IEEE Spectrum: Technology, Engineering, and Science News. https://spectrum.ieee.org/</a>
<a name="ref-7" >Ihaka, R. (1998, May). Past and Future History. R. https://cran.r-project.org/doc/html/interface98-paper/paper.html</a>
<a name="ref-8" >Lam, J. (2019, March 20). Introducing R Tools for Visual Studio. Visual Studio Blog. https://devblogs.microsoft.com/visualstudio/introducing-r-tools-for-visual-studio-3/</a>
<a name="ref-9" >Marks, R. A., Vieira, D. K., Guterres, M. V., Oliveira, P. A., Fonte Boa, M. C., & Vilela Neto, O. P. (2021). Design and Test of Digital Logic DNA Systems. IEEE Design & Test, 1–1. https://doi.org/10.1109/mdat.2021.3069369 </a>
<a name="ref-10" >Project Jupyter. (n.d.). Project Jupyter. https://jupyter.org/</a>
<a name="ref-11" >The R Foundation. (n.d.). R Project Contributors. R Contributors. https://www.r-project.org/contributors.html</a>
<a name="ref-12" >Reades, J. (2020). Teaching on Jupyter. REGION, 7(1), 21–34. https://doi.org/10.18335/region.v7i1.282</a>
<a name="ref-13" >RStudip. (2021). RStudio. https://www.rstudio.com/products/rstudio/</a>
<a name="ref-14" >Sanghar, M., Shukla, V. K., Verma, A., & Sharma, P. (2021). Implementation of Support Vector Machines Algorithm through R-Language for Diabetes Database Testing. 2021 11th International Conference on Cloud Computing, Data Science & Engineering (Confluence). https://doi.org/10.1109/confluence51648.2021.9377124</a>
<a name="ref-15" >Stack Overflow. (2021). Stack Overflow Developer Survey 2020. Stack Overflow Developer Survey. https://insights.stackoverflow.com/survey/2020</a>
<a name="ref-16" >Vance, A. (2009, January 7). Data Analysts Captivated by R's Power. The New York Times. https://www.nytimes.com/2009/01/07/technology/business-computing/07program.html</a>
<a name="ref-17" >Varghese, L. S. (2021, April 9). Netflix Shows in R_CW. Kaggle. https://www.kaggle.com/lisasvarghese2037041/netflix-shows-in-r-cw</a>
<a name="ref-18" >Venables, B., Smtih, D., & The R Core Team. (2021, April 1). An Introduction to R. Comprehensive R Archive Network. https://cran.r-project.org/doc/manuals/r-release/R-intro.pdf</a>

LastFM Now Playing/Server Heartbeat

Fri, 18 Mar 2022 04:00:00 GMT

Disclaimer up front. This is 100% based on the Raspberry Pi now playing from chorus.fm. However, instead of making a single page PHP application, written in Python using the Flask framework. I don't currently have a raspberry pi running this 24/7, but that is something I'd like to do for a future project

App Creation

last.fm API

Step 1 is to get an API key. This is normally very easy and the API for last.fm is free (as far as I have seen). The end goal is to have a very low request volume application. I am going to update my Now Playing screen every 15 seconds and the Stats screen every 30 or 60 seconds. This isn't a real time playback system so a slower update rate is fine with me. I may do similar to the original project and set up some shortcuts for whatever display solution I end up with.

Basic Flask System

While this is perfectly acceptable as a single page system, I wanted to actually make a small Flask site so that I can extend functions beyond just the now playing screen if I want. Thankfully, the philosophy of flask is to add on items as needed so with an application this simple, it is very light weight. The basic setup for the project is:

app
├── app.py
├── classes
├── requirements.txt
├── static
└── templates
    └── Components

app.py

app.py is the main entry point of the project when running with a python command. it will contain the if __name__ == __main__ check that will determine if the file is being run directly or loaded from an external program.

The basic Flask setup is

from flask import Flask
app = Flask(__name__)

@app.route('/')
def index():
    return '<h1>Hello!</h1>'

if __name__ == "__main__":
    app.run(host="0.0.0.0", port=8080, debug=True)

The two most important parts of this basic program are app = Flask(name) which actually imports associates the Flask server with the program and app.run(host="0.0.0.0", port=8080, debug=True) the host='0.0.0.0' allows external machines to access the flask test server. If you are only running Flask locally, you don't need this value. I chose to run the program during development on a virtual machine running Ubuntu. This will end up being the operating system I run this project on (spoiler: I'm using it as a health check like Jason Tate as well)

Now Playing

The now playing portion is simple enough. To follow Don't Repeat Yourself (DRY) code I made a few helper functions for working with last.fm. I made class LastFmApi that utilizes the requests module for HTTP get operations and contains utilities to simplify the returned JSON from last.fm into only the parts I care about for my now playing screen. I could have saved some complexity by just throwing all this data into a python dictionary, but then I would lose intellisense when utilizing the information elsewhere in my project. Because of this I decided to make a class with methods for getting/setting artist info and images and return an array of these objects from my last.fm calls. I am still learning Front-End Development so for the styling I just stole the CSS from the inspiration project here I like the general look of it good enough for now and can modify/extend as I slowly improve upon this implementation

Stats

One thing I found missing from the original project was the idle screen was a little lacking from what last.fm can give you through the API. The largest deviation I have from the original is I modified the stats screen so that I retrieve a random set of stats when a song isn't currently playing. To accomplish this I made to lists of sets, one for a time frame and one for the stats. I decided to use two lists because then I can operate the stat pulled and the time frame with their own randomness as well as keep how many records to pull for each stat (e.g. it is easier to fit more top albums on the stats page that top artist info). My goal is that I can also extend this to display top tracks and top tags from my account as well.

time_frames = [("All Time", "overall"), ("7 Days", "7day"), ("3 Months", "3month"), ("Month", "1month"), ("6 Months", "6month"), ("12 Months", "12month")]

stats = [("Top Albums", 12), ("Top Artists", 6)]

def get_top_section() -> tuple:
    data = None 
    time = random.randint(0, (len(time_frames) - 1))
    stat = random.randint(0, (len(stats) - 1))
    stat_type = stats[stat][0], f' - Last {time_frames[time][0]}' if time > 0 else ' - All Time'
    
    logger.write_log(message=f'Retrieving {"".join(stat_type)}')

    if stat == 0:
        data = last.process_albums(last.user_get_top_albums(period=time_frames[time][1], limit=stats[stat][1]))
    elif stat == 1:
        data = last.process_artists(last.user_get_top_artists(period=time_frames[time][1], limit=stats[stat][1]))
    else:
        raise ValueError(stat)

    return stat_type, data

For the MVP I am only handling Artist and Album data, but the logic will be easy enough to extend to tags and tracks once I get a chance. After I pull the stats I had to modify the logic for how the data is displayed depending on the stat pulled which was easy enough with flask's if/else logic

<caption>Screen shown when a song is playing</caption>

<caption>LastFM Top Albums stats page</caption>

<caption>LastFM Top Artists stats page</caption>

App Deployment

Now came the fuzzier bit for me. I know how to complete simple deployments of sites and how to run backend services. So I figured now would be a great time to challenge myself and learn how to deploy my app as a custom docker image to Azure

Docker

Docker containers have become one of the largest skills in software in recent years. The basic idea is to make an image that is self contained for a service and use orchestration through Kubernetes or Docker Swarm to make an application that is resilient and can auto scale for the needs of your application. I am not good enough (yet) at Docker to go into the details of how docker works and the lifecycle. I can make an image of my current app and it is stored in a public docker hub I haven't made a readme file yet but it can be pulled with docker pull corycarte/nowplaying. Right now I have the image with my API key and username, this is easy enough to remedy with a bind mount to where the image is run. I would want to document that in the README before completing that change though. One last annoyance of note. When I graduated last April with my Bachelor's, I treated myself and bought a fancy new M1 Macbook Pro, when building my docker images with it it builds the ARM version. This version doesn't run on the server I want to deploy it to so I have to make sure I tag the version as ARM and build a separate version for other machines. I am sure there is a remedy to this, I just haven't researched it yet (topic for the future).

Deployment Option 1: Azure Web App Service

The deployment option for the future is to run this as a scalable microservice in the cloud. I chose to attempt Azure since I use it at work anyway so let's try and get it running in their Web App Service.

Step One: Azure Resource Group

Azure resources are contained within resource groups, since this is a test only I made a specific test-rg resource group to hold my application. The steps are very straight-forward and are detailed here

Step Two: Azure container registry

Azure like all the cloud services has their own private container repos. These can be used as a part of a CI/CD pipeline for docker containers so let's go ahead and set one up. Similar to the resource group. Straightforward to set up, detailed here

Step Three: Web App

The final step is to actually get your container up and running. I followed all the steps here and... well it never worked for me. I also went through with the demo and it still gave me a 500 status when attempting to navigate to the deployed app. This wasn't the way I wanted to deploy this since I want to keep it as a heartbeat on a server so the Azure Web App not working isn't the end of the world.

Deployment Option 2: Nginx

Keeping in fully ripping off chrous.fm, I can deploy this container image onto an existing server and use it as a health check. This for sure maintains more of a pet than cattle relationship with the given server. I am not building a Highly Available service and caring over a couple of servers that run various personal projects isn't that much of a hassle. This approach (at least the way I want to do it) does require some updates to my webserver. Currently Nginx is the server of choice for small scale projects in lieu of the Apache Http Server. This isn't to say that one is superior, I just want to us Nginx for this. Since I am deploying a Docker container, I will expose a port on the server to get my HTTP Request into the container. I will accomplish the routing of this request from the open web through Nginx by setting up proxy pass and a rewrite rule. Something fun I dealt with (and it may be due to being new with Nginx) is though the proxy worked properly for my flask app and css files, the base.js file I made wasn't being properly retrieved. Not too big of a deal since all I am using that file for is to refresh the page periodically so instead of solving this issue, I just added a script tag in my base.html template with the refresh function. Is this the best solution, no. However, it works.

Conclusion

Overall, simple to get up and running on my server to keep an eye on it. The main thing I want to do in the future is to modify the CSS to make it more of my own project vs a (mostly) complete ripoff project. I've used Flask minimally in the past though I was still able to learn about Jinja macros and Nginx reverse proxy into Docker containers. The project code is up on my github now

Learning iOS and Swift

Wed, 23 Mar 2022 04:00:00 GMT

With an average iOS developer salary at 98k a year an early career engineer with a MacBook could definitely be drawn to the world of mobile development. (though Glassdoor reports the average Android developer makes roughly the same at 99k per year).

Why iOS?

Why should I be learning iOS itself instead of learning a technology for cross platform development like React Native or Xamarin. I work full time right now as a full stack developer using C#, don't I want to keep down that path? The main advantage of learning native iOS development is two fold for me.

Using Swift and Swift UI enables the development of iOS, iPad OS, and MacOS applications with a more unified code base
the common perception is that native applications perform better.

While I don't have specifics, and I think for most applications, a shared code base for android and iOS would perform adequately for a lot of applications. A large amount of people like to point to the AirBnB medium post for stopping development with React Native as to why native mobile development is the best way to make mobile applications. Personally, I am not as sold on point 2 as I am on point 1.

Ok, fine, but why learn Swift?

Swift is a language developed by Apple for interoperability with Objective C originally announced in 2014 aside for a small period when Swift was being pushed for use with TensorFlow no organization outside of Apple is using Swift right now. I am choosing to learn Swift purely because it is the more modern language and then hopefully bright future of SwiftUI.

How to Learn a language?

Learning a new programming language is a constant within Software Engineering. To be an efficient engineer today requires the ability to efficiently learn new languages there are many approaches:

Leetcode problems
Small projects
Courses/tutorials

I tend to work best with option 2 or 3 so that’s what I’m going with.

My First steps

I'm sure a lot of people are like me, I learn by doing. So the best way to begin my journey into iOS development was through the The AppBrewery and their Complete iOS and Swift Boot Camp. Not only does this program give an intro to mobile development, it also covers basics of the Swift language, and the Model View Controller (MVC) design pattern (the version I purchased also includes app design and app marketing courses). One of the major advantages to a course like this for me is that I learn the best when developing actual projects, but I am too much of a perfectionist to make half-formed applications for the sheer purpose of learning individual pieces of the whole that is iOS development. I purchased the Udemy course, it was on sale so thankfully I was able to get it all for only $20. Once I am done with the course I can move on to creating large scale projects for my own use and hopefully create something that could generate revenue.

Intrinsic Motivation in Software Engineering

Sat, 08 Jul 2023 04:00:00 GMT

Software engineering is a career that requires constant Intrinsic Motivation. Full Stop.

A lot of jobs outside of software will require some Intrinsic Motivation upfront to get to the point at which an individual can be motivated by something like a paycheck. Software engineering is a field where intrinsic motivation doesn't stop once you start getting paid to be a professional in the field. This is a field where employers may look at you in a different light if you aren't working on personal development projects. This is also a field that requires individuals to spend unpaid time to continue to learn about developments within the field. I haven't worked within another field that is constantly evolving so I could be incorrect about how unique the tech field is to be relevant I not only need to work professionally I also need to use my time to learn and even develop projects.

The hardest part of all of this is to be able to stay properly up to date without getting stuck with shiny object syndrome (something I have struggle with in the past). There is a fine line we have to hit between staying current and bouncing around between any of the new javascript frameworks that are coming out and never actually progressing in them technically.

I have been slowly learning iOS Development, and the biggest thing I have learned about it is that progress is a requirement. This can be the introduction of new tools and libraries, like Android Jetpack or SwiftUI. This can even mean learning new languages with Android's shift from Java to be Kotlin first and iOS's shift from Objective-C to Swift. Even if I didn't want to learn the native tools, I could have learned a new way to use a language I already know to build apps with tools like React Native to build apps with Javascript or .NET MAUI to use C#. Change is hard, and change in technology is harder at the pace it changes. Just from when I first learned about software development as a career in 2010 to getting my first developer job in 2019, it feels like the entire internet changed (ReactJS was released in 2013).

Now I am not here to say that the only way to have a career in software is to constantly. I worked with a senior engineer in the past that had no interest in new technologies and new methods. This isn't me trying to say he wasn't intelligent and didn't work hard. He was extremely smart and an incredibly hard worker gladly putting in 10-12 hour days. I learned a lot from him (mainly the magic of sed and awk) that are invaluable skills college didn't teach me and bootcamps probably don't teach. However, with all of that motivation to work he had no interest in working with web technologies or moving past some of his UNIX commands. He was very set in his ways and very comfortable in his position (when I was working with him, he had been in that position for almost 20 years). It was a position that allowed him to not continue evolving with technology and when faced with using some newer technologies for a specific task, he refused to learn them. If that is what someone wants to do in this career, then that is their choice and is a way to approach this career also. Those jobs exist and no one can judge the developer that wants to work 9-5 and not think about software between the stop of one work day and the start of the next.

The Journey for My Second Brain

Thu, 15 Feb 2024 00:00:00 GMT

As I get older, I find myself thinking more about knowledge management and the concept of a second brain comes up

David Allan (Author of Getting Things Done says "Your mind is for having ideas, not holding them". Your brain not suited for storage, it is more adept to processing information and developing new ideas.

There is so much that I have learned throughout my life, and the best way I've heard it put is "My brain is a dumbass, I'm not going to remember this" -Ali Abdaal

Because of my desire to not forget bits of information. I have stumbled across the ideas of Knowledge Management with systems like Zettelkasten, and more recently the concept of the Second Brain. Though with the current state of my notes and knowledge, that presents a few problems.

What is Wrong With Me

Problem 1: Notes spread throughout systems

To put it simply, I have too many notes spread in too many systems. I have notes in Obsidian, Notion, Apple Notes, and random text documents. I don't know if this problem is specific to my personality and brain, or if this is a widespread issue and it just doesn't seem like it due to some strange form of Survivorship Bias for productivity.

It doesn't help that the well formed systems one can be exposed to through professional productive people like Thomas Frank and Ali Abdal fall into this productivity survivor camp. They share their systems because their systems have helped them build a platform around which to share productivity advice.

Problem 2: Inconsistency

If TikTok has taught me anything, it is either that I have Attention Defecit Disorder (ADD) (note: obviously self-diagnosis is a bad thing, don't do it) or I share a lot of traits with someone with ADD. This leads to major errors in my consistency on any system I attempt to implement.

Problem 3: Lack of a Primary Productivity System

Somewhat related to problems 1 and 2, I currently do not follow any particular system for productivity. I have looked at Getting Things Done and Time Blocking and I am currently attempting to following some hybrid mix of the two. Really heavily on Google Calendar and Todoist to keep track of events and tasks while being cross compatible with Windows and MacOS.

Problem 4: Lack of Focus

This one is pretty personal to me. I spent so long focused on one major goal, finishing college, and thankfully April 2021 I was able to realize that goal and graduate with my Bachelor's in Computer Science. Because of life circumstances this wasn't a straightforward goal and instead of the normal 4 year college track, it took me a total of 14 years to complete a Bachelor's degree. Finally achieving a goal that you have worked towards for so long is hard to take in, and the achievement of that goal has left me like a ship rudderless. The productivity "systems" I had in place were only suitable for school, they were intense focus for really short bursts (about a semester's worth, weird), and were highly dependent on the singular goal of "a degree" and driven by the tasks of each class taken. Now that I have finished college (Woo!), I can start living my real life. The only problem is, what do I do now?

Moving Forward: Where am I Now?

I will be really honest, I started writing this almost a year and a half ago and I just opened it back up and realized I should probably actually finish it out. I am not that much closer to solving any of the 4 problems above, but I still try every day. I tried Bullet Journals and other paper methods thinking that my problem was throwing all my stuff into electronic systems and maybe I have some for of lacking object permanence with electronic files. I don't think I am that much closer to solving my second brain struggles, but I still try every day.

WWDC 2025

Thu, 12 Jun 2025 00:00:00 GMT

For the last few years, I have been slowly learning ios, but I haven't gone all in to becoming an iOS developer by any means. There are some nice things about working with Swift as a language, and some very fun parts about MacOS/iOS development (not XCode). My personal laptop (probably in need of a refresh by now) has been a MacBook pro since I graduated college in 2021.

However, I have always just been a very passive viewer of WWDC. I got more involved watching the keynotes live (or close to live) most likely because of listening to the Connected podcast, and I would even say both Stephen Hacket and Federico Viticci have in some ways also inspired me to become more of a writer (I'm still not great at it, but here we are).

I think I am going to follow inspiration from another creator (Elliot from Dreams of Code) for all side projects I work on. He made a commitment to using Go as much as he can, so for the next 6 months or so I will write in Swift unless it isn't possible. I have had a few ideas for little apps, and I have been curious about things like Vapor for running Swift on the server. So why not go all in now?

The main reason for wanting to write something (somewhat) timely is of course, Apple's World Wide Developer Conference (WWDC).

I haven't watched through the technical sessions yet, but I did watch the keynote. While overall, there are some minor (but possibly very good) improvements to some of apple's built in software. The biggest thing people will talk about is going to be Liquid Glass. I have seen plenty of hate towards the aesthetic even from the single developer beta being released (looking at you r/ios). When I look at it, I see a strange form of retro-futurism. The new UI somehow has a look and feel the late 90s but also modern. Due to some life circumstances, I will be spending more of this week at home, and with that I think I am going to try to attend more of the WWDC things. I don't think I will commit to a lab or the hands-on items, but definitely watching a bit more about Liquid Glass and the new OS releases is more feasible to me than in the past.

I always install the developer beta (on my daily use phone. Sorry literally everyone in my life) and then move to the public beta when it’s out and with this ui overhaul and the fact that I’m still using the iPhone 13Pro I have to say how impressed I am with the performance even with dev beta 1. I have two main opinions for liquid glass, but the primary one is that I enjoy it. It’s almost the same joy I felt with the Dynamic Island announcement and while there will be plenty of issues and kinks to work out. It’s already nice to look at. Though iOS has always had (in my opinion) the problem of being too wrapped around animations and transitions providing aesthetics first. Which I can appreciate in some ways.