Refactoring in Python is not that hard

In this article, I will walk you through:

1. The signs of need for refactoring.
2. What is considered refactoring?
3. What to do before refactoring?
4. Step by step guide to restructuring Python package

The examples are written specifically with Python in mind. However, the general principles shall hold true across all languages.

The signs of need for refactoring

1. Your versions are no longer supported. The best written code will still be obsolete after some time.
2. You want the new features in newer versions of frameworks, libraries or even programming language.
3. Your code was written quick and dirty with tons of technical debts.
4. Your code is hard to navigate. Countless indirect access.
5. Not comply with standards, style guides, conventions.

What is considered as refactoring?

The term refactor, covers a broad array of actions and definitions. While the term itself generally lead to a common goal, which is a cleaner, better code base, there are still a lot of actions can be considered as ‘refactoring’.

1. Upgrade your dependencies to newer version
2. Rename your code functions, classes, modules, etc
3. Reorganize your code base, moving a function from a file to another
4. Improve implementation of a function to increase performance
5. Reformat your code to make it standards compliant

Before Refactoring

In spite of distinction from one programming language to another, there are generally some steps and principles to be held while refactoring.

Step 1: Get A Glimpse of Your Code Quality.

Run static analysis tools. Static analysis tools gives you a summary report with quantitative statistics that can be compared while refactoring from time to time.

In Python, I personally use Prospector, a static analysis suite encompasses other tools such as pep8, Pylint, etc. The best thing I like about Prospector is its ability to detect and adapt to the frameworks I use (Django, Celery).

There are 3000+ messages found. The messages found should be vastly reduced after refactoring.

Do note that not all messages found are valid, there is a chance of false positive.

Step 2: Prepare and Verify Test Cases

Code without test is bad code by design. Do you have test cases in place? If so, how complete are they? Are those test cases up to date?

Why do we need test cases before refactoring? You can argue that you won’t need any, even if you are doing a simple change such as adding validation. Well, trust me, you gonna get yourself caught by unexpected behavior responded by the masterpiece you touched up.

I’m not gonna convince you the importance of having test codes in general. Test code before refactoring is to ensure the behavior of your system is consistent after refactoring.

Even if there are test cases in place, which gives you a green-light, you should still verify the test code. Let me tell you why.

Imagine you run whatever tests available before you attempt to refactor, and you get the following results.

Tests all passed. Nope, they look like all passed.

However, when you dig further, you found this test case.

So, you see the point of verifying it?

Besides, test cases are often the best documentation for a software. While navigating within codes, they are also your best GPS navigator.

Start Refactoring — Reorganizing / Restructuring

In this section, I will walk you through an example config.py by reorganizing the structure, merging duplicated methods, decomposing and writing test code to ensure backward compatibility.

config.py looks like this:

Step 1: Write Backward Compatible Code

This step is crucial. Before refactoring our code, test cases MUST be in place. In this case, we write backward compatible code to ensure all references to the classes/functions/constants are still working.

In __init__.py, we shall redefine the class/method signatures:

The __init__.py is incomplete for now. We will revisit the file later. Next, we write a test case to make sure we can still import the package as if we are importing the old package.

This is a simple test case, you may notice some backward compatibility issues are not caught in the test case.

Step 2: Reorganizing Package Structure

This section gives you an idea on how you can reorganize your Python package. Let’s revisit the config.py we have:

Can you spot what’s wrong here? It is messy, there are constants, helpers, duplicated codes in a single file. When the code in config.py grows larger, it will become increasing difficult to navigate within. With this messy structure, you are breeding a spot for circular dependency, hidden coupling and refining the recipe for the tastiest spaghetti code.

How can you reorganize config.py? To me, separation of concerns comes across my mind. The following structure is often considered a good practice to structure Python package (this structure is used in Django as well).

config/
├── abstracts.py    # All the abstract classes should live here
├── constants.py    # All the constants should live here
├── exceptions.py   # All custom exceptions should live here
├── helpers.py      # All helpers should live here
├── __init__.py     # All backward compatible code in here
├── mixins.py       # All mixins goes to here
├── serializers.py  # All common serializers goes to here
└── tests.py        # All `config` related tests should live here

Let’s revisit our config.py before refactoring and identify where the individual piece of code should reside.

After refactoring, config.py should become a Python package config with a __init__.py in it.

utils/
├──config.py        # To be removed
└──config/
├── constants.py   
├── helpers.py   
├── __init__.py  
└── tests.py

In utils.config.constants:

In utils.config.helpers:

Step 3: Eliminate and Merging Duplicates

In utils.config.helpers, there are 2 similar methods/functions get_logging_level() and ConfigHelper()._get_logging_level(). Assuming both implementations are identical, it means we have to find a best place to host the function.

In this case, I remove the standalone get_logging_level() and keep the one in ConfigHelper.

Hierarchy of Decomposed ConfigHelper

We host our AbstractBaseConfigHelper in abstracts.py:

In mixins.py:

In helpers.py:

ConfigHelper is now decomposed into multiple classes and mixins.

Step 5: Complete Our Backward Compatibility Code

In Step 1, we added some code in __init__.py. However, it is largely incomplete. Let’s revisit the file:

Notice that the bridge between the code above and our newly organized config package is still missing. To establish the bridge, we edit our __init__.py into:

Step 6: Notify The Developer

Up to Step 5, our config is properly refactored. However, we need to keep the developers notified about the change. Is there any straightforward way? Yes. We can emit a warning message whenever a developer is trying to import an obsolete function/class/method. For example, we annotate the old functions/classes/methods with decorators:

In our __init__.py, we add decorator like this:

After Restructuring

After restructuring our Python package, we run our test case and make sure it’s all passed.

Conclusion

Up to this point, you should be able to understand the quality of your code base, understand the concept of refactoring, identify the need of refactoring, and understand how can one restructure/reorganize a Python package.

First published on 2018-11-20

Republished on Hackernoon